The geospatial PDF must be properly and accurately georeferenced and comply with the Adobe Acrobat PDF 1.7 specification (such as maps produced using MAPublisher or ArcGIS). Also, in Adobe Acrobat, use Advanced > PDF Optimizer to create a balance of small file size and good quality. Large rasters in the map could increase file size significantly, so try to use optimized/compressed images and only when necessary.
GeoTIFF images can be optimized by converting them to Indexed color using raster software like Adobe Photoshop. Image resolution should range from 200 dpi to 300 dpi to achieve sharp images suitable for multiple zoom levels and should be compressed using LZW compression. Note that higher resolution images can have more zoom levels, but create larger sized files.