Multi-cylinder, small-displacement, designs were in the European tradition--and BRM's. Three liters was in Ferrari's wheelhouse: Enzo had been doing V-12's
around that size for fifteen years. A 16 could make more power, in theory.
But a V-16 was too long for the small monocoque chassis by then dominating Formula 1. Why not stack two flat-8's, one atop the other? Along with other engine makers, BRM was late to the party in 1966. Jack Brabham stole a march on them by using a production-based Repco V-8 in his own chassis to win the championship.
Everyone was ready for 1967: Ferrari's V-12, the BRM H-16, and the Cosworth V-8 in the back of the Lotus 49. The latter design was revolutionary:
4 valves per cylinder at a narrow angle, which minimized the size of the heads, with its ancillary parts alongside the sump. And it was a load-bearing chassis member. This lowered the center of gravity and the weight of the car. It was remarkably compact, thus keeping the engine and its accessories out of the air stream. The Cosworth proved to be the solution to the 3-liter rules. By the early 1970's, everyone but Ferrari was running a Cosworth in his own chassis. It was dethroned only when Renault figured out how to make a 1.5-liter turbo produce more power and last the length of a Grand Prix while doing it.
What were the problems with the H-16? It was massive for a 3-liter, and its center of gravity was high because of the "dead space" between the crankshafts. The 90-degree bend between the intake and exhaust ports was less efficient than a crossflow cylinder head (the H-16 never produced the power it was theoretically capable of). Single-plane 8-cylinder cranks, whether for a flat or a V configuration, are notorious for secondary harmonic vibration. Porsche's flat 8's and Cosworth's V-8 both suffered from this. BRM's solution, gearing the cranks together in 90-degree phase to each other, did not solve the problem. (Porsche and Cosworth both eventually made conventional crankshafts, with throws and cylinder firing order optimized for smoothness.)
There is a saying, "If it looks right, it is right." While the saying isn't always right, it's opposite almost always is: "If it looks wrong, it probably is." You can see the elegance of the Delage V-12 in my previous post. You can see the complexity to no purpose in the BRM H-16. It lasted only one season and was replaced with a V-12. Here's a video with audio (sounds a bit like a big-bore V-8 at low revs...)