Are Libertarians inherently anarchists or just labeled that way by our overly large government?

Left Libertarianism is social anarchism, but other than that, no. Those who claim they are Anarchists are merely those who want large government controlling a part of their lives (whether it be a social issue with the right, or a fiscal one with the left).