Preheat oven to 425 degrees.
Layer half of the lemon slices and half of the rosemary leaves on bottom of a rectangular baking dish. Scatter half of the olives and half of the garlic around the pan.
Season both sides of the fish with salt and pepper and lay (skin-side down, if applicable) on bed of lemons, rosemary, olives and garlic. Place remaining rosemary, garlic and olives on top of fish and top with remaining lemon slices. Drizzle with olive oil and freshly squeezed lemon juice.
Bake for 15 to 20 minutes or until fish flakes easily (depending on thickness of fillets).