The memory elements have been the research hot spot for a long time. However, there is a litter works on the linear memory element. This paper presents a study of a new memory system containing a linear memory element. The study shows that the system not only has two kinds of route to hyperchaos, but also exists many kinds of coexisting attractors in the phase space. Moreover, the system can generate more complex hyperchaotic behaviors. To prove it, we find a new kind of topological horseshoes with two-directional expansions that consist of three disconnected compact sets. This new kind of horseshoes suggests that the topological entropy of the hyperchaotic attractor is larger than other hyperchaotic attractors reported before. For detailed study of the hyperchaotic invariant set, we also demonstrate a method to extract the orbits from the hyperchaotic horseshoes.