In a mixing bowl, beat cream cheese until smooth. Add eggs, one at a time, beating well after each addition. Add flour; mix well. Stir in potatoes, Swiss cheese, onion, salt, pepper and cayenne pepper.
In a large skillet, heat butter and oil over medium heat. Drop batter by 1⁄4 cupfuls; press lightly to flatten. Fry until golden and crisp, about 5 minutes on each side.
Drain on paper towels.