In 1759, Captain James Potter, accomained by William Thompson and an Indian guide, came upon Penns Valley while scouting areas suitable for stockade forts. It would be many years until he would return and build these forts.

But return he did, and in 1774 built the first of three stockades at Old Fort. The second was to be built near what is now Penn Hall, and the third to be built near what is now Woodward. However these forts didn't help much against the raiding Indians of the time. In 1778, in what has become known as the Great Runaway, almost all of the residence of Penns Valley, including Potter himself, fled the valley to safer havens to the south.

General Potter did manage to make it back to the valley prior to his death in 1789. He sold some of his land, but still had over 7000 acres to his name at the time of his death. His daughter to his first wife married James Poe for which Poe Valley is named. A daughter to his second wife, Martha, married Andrew Gregg for which the township gets its name.

James Potter's only son, James Potter Jr., inherited the majority of his father's estate. He himself became a general in the newly formed United State militia, James Potter Jr. settled and maintained his residence in what is now Potters Mill.