Nature uses hierarchical fibrillar structures to mediate temporary adhesion to arbitrary substrates. Such structures provide high compliance such that the flat fibril tips can be better positioned with respect to asperities of a wavy rough substrate. We investigated the buckling and adhesion of hierarchically structured adhesives in contact with flat smooth, flat rough and wavy rough substrates. A macroscopic model for the structural adhesive was fabricated by molding polydimethylsiloxane into pillars of diameter in the range of 0.3–4.8 mm, with up to three different hierarchy levels. Both flat-ended and mushroom-shaped hierarchical samples buckled at preloads one quarter that of the single level structures. We explain this behavior by a change in the buckling mode; buckling leads to a loss of contact and diminishes adhesion. Our results indicate that hierarchical structures can have a strong influence on the degree of adhesion on both flat and wavy substrates. Strategies are discussed that achieve highly compliant substrates which adhere to rough substrates.