Hey,
its nice to see that someone is running into the same issue we are currently looking in. Did you tried to read/write data/parameters to the AD9361 via iio-oscilloscope? That should not work as well right? What you could do is to set parameters via the linux driver (AD9361 high performance, highly integrated RF Agile Transceiver™ Linux device driver [Analog Devices Wiki]). Maybe one of the specialist could give some informations how the achive the same support (matlab/iio-scope libiio etc) with a costom ad9361 design, refer to FMCOMMS2/3-EZB without FMC Connector
Regards
Jan