In this brief, a CPU-FPGA based heterogeneous acceleration system and a subgraph segmentation scheme for CNN-RNN hybrid neural networks are proposed. Winograd algorithm is applied to accelerate CNN.