Human settlement in the area can be traced back to approximately 400 CE, when Native Americans forged three trails to better access the area's freshwater springs. In the 16th century, the Creek Muskogee tribe settled the area, where they remained until the early 1800s, when they were forced out of the area due to the discovery of gold.

In 1821, the federal government held a number of land lotteries in the area, resulting in the purchase of land in present-day Sandy Springs and its subsequent settlement. The Austin-Johnson House, the oldest existing unaltered house, was built in 1842 on what is now Johnson Ferry Road. In 1851, Wilson Spruill donated 5 acres (2.0 ha) of land for the founding of Sandy Springs United Methodist Church, near the natural spring for which the city is named. In 1905, the Hammond School was built at Johnson Ferry Road and Mt. Vernon Highway, across the street from the church.

In 1950, the state legislature blocked Atlanta from annexing the community, which remained rural until the Interstate Highway System was authorized by the Federal-Aid Highway Act of 1956. In 1959, after a fire at Hammond Elementary School, William Hartsfield, the mayor of Atlanta, urged residents to support annexation so that the area would have better firefighting protection. Community opposition killed the proposal. In the early 1960s, Georgia 400 and Interstate 285 were constructed, connecting Sandy Springs to metro Atlanta and initiating a housing boom that brought new residents and major land development as part of the white flight from Atlanta after the Civil Rights Movement won greater racial integration within Atlanta.

In 1965, Hartsfield once again proposed the annexation of the Sandy Springs area. Spokesmen for Sandy Springs promised residents to "build up a city separate from Atlanta and your Negroes and forbid any Negroes to buy, or own, or live within our limits" should they reject annexation. In 1966, annexation by Atlanta was defeated in a referendum, with two-thirds voting against.

Efforts to incorporate Sandy Springs began in 1966 in response to attempts by the city of Atlanta to annex this unincorporated area of north Fulton County.

In the early 1970s, the city of Atlanta attempted to use a state law to force annexation of Sandy Springs, which failed after the Supreme Court of Georgia ruled that the law was unconstitutional. In response, a group of residents formed the Committee for Sandy Springs 1975 to lobby for the incorporation of Sandy Springs.

During this time, proponents for an incorporated Sandy Springs argued that their taxes were disproportionately going to other, largely non-white, communities in Fulton County. In every legislative session, state legislators representing the area introduced a bill in the Georgia General Assembly to authorize a referendum on incorporation. Legislators representing Atlanta and southwestern Fulton County, who feared that tax revenue would be lost from incorporation, blocked the bills, using the procedural requirement that all local legislation be approved first by a delegation of representatives from the affected area.

In 1991, the Georgia state government determined that Sandy Springs, along with other wealthier, and predominantly white, communities in Fulton County was being taxed below statewide minimums, resulting in an increase in taxes for the area. Some Sandy Springs residents, including Mitch Skandalakis, launched a number of campaigns against the taxes, and launched an unsuccessful lawsuit against the state.

On January 16, 1997, Eric Rudolph bombed an abortion clinic in Sandy Springs.

When the Republican Party gained a majority in both houses of the Georgia General Assembly in 2005, the procedural rules previously used to prevent a vote by the full chamber were changed so that the bill was handled as a state bill and not as a local bill. The assembly also repealed the requirement that new cities must be at least 3 miles (4.8 km) from existing cities that had stymied previous attempts to incorporate due to Sandy Springs directly bordering both Roswell and Atlanta. The bill allowing for a referendum on incorporation was introduced and passed as HB 37. The referendum initiative was approved by the Assembly and signed by Governor Sonny Perdue.

A referendum was held on June 21, 2005, and residents voted 94% in favor of incorporation. In November 2005, voters returned to the polls to elect a mayor and six city council members.

Formal incorporation occurred on December 1, making Sandy Springs the third-largest city ever to incorporate in the U.S. The city's police force and fire department began service in 2006. Upon incorporation, Sandy Springs initiated a nontraditional approach by operating as a public-private partnership (PPP), with all but six full-time employees being contracted.

In 2010, the city undertook a procurement process to rebid all general city services, which was won by CH2M Hill. The timing of this contract, during the Great Recession, allowed the city to leverage a cheaper contract due to the economic downturn.

In 2010, the city became the first jurisdiction in Georgia to successfully "bail out" from the preclearance requirements of Section 5 of the Voting Rights Act.

In 2019, the Sandy Springs City Council moved to scale back the PPP model, directly hiring 183 contract employees, leaving only 15 outsourced full-time workers by the end of 2019. The city will still outsource a number of services, including the city attorney's office, as well as security, street sweeping and ambulance services. The move was expected to save $2.7 million in the next year and more than $14 million over 5 years.

The boundaries of Sandy Springs are Atlanta to the south, Cobb County (at the Chattahoochee River) to the west and north, Roswell (also at the river) to the north, and Dunwoody and Brookhaven, at the DeKalb County line, to the east. A small panhandle in the northeast extends between the Chattahoochee River to the north and Dunwoody to the south, ending in a very small border with Peachtree Corners in the extreme western edge of Gwinnett County.

Sandy Springs has a humid subtropical climate (Köppen climate classification Cfa). During January and February 2014, the Atlanta area, including Sandy Springs, experienced a severe snow storm and a severe ice storm, both of which left much of the region without power, caused major travel disruptions, and the former storm forced people to take shelter in cars and schools as the city was underprepared for the black ice that prevented transport.

City Springs, the downtown district of Sandy Springs, is usually defined as the area to the south and east of Sandy Springs Circle, to the north of Interstate 285, and to the west of Boylston Drive. It is located approximately 12 miles directly north of Downtown Atlanta. In the absence of a traditional downtown, city leaders created City Springs, a multi-use development containing municipal offices, residential, retail, green space and a performing arts center, landmarking a formal "downtown" for its residents. City leaders purchased the property in 2008, which was once the site of a former Target (formerly Richway) shopping center, located between Roswell Road, Johnson Ferry Road, and Mount Vernon Highway. Since then, the surrounding area in the district has become a center for urban renewal for the city, with many new mixed-use apartment developments being planned or built, primarily replacing old strip malls along Roswell Road. The City Springs center officially opened in 2018, 10 years after the original site purchase. The official address for the complex is on Galambos Way, named after the city's first mayor, Eva Galambos. Within the City Springs district is Heritage Green, which is home to the spring which spurred the name of the city.

Riverside is the western district of the city, located south of Dalyrmple Road and west of Roswell Road, bordering the Chattahoochee River to the west, forming the western border with Cobb County. It is a high-income, residential area marked by winding, hilly roads and old growth forest. The main roads are Heards Ferry Road and Riverside Drive, and it is located off the Riverside Drive exit of I-285. Two of the public schools within Sandy Springs are located here, Heards Ferry Elementary and Riverwood International Charter School. The headquarters for the Fulton County Board of Education are also found in this district. Many of the neighborhoods in this area derive their name from the river.

The Dunwoody Panhandle, or just "The Panhandle" is a residential area bounded by the Dunwoody city limit to the south, the Chattahoochee River to the north, Georgia 400 to the west, and Peachtree Corners city limit to the east. The district's name is derived from the fact that it is wedged between the river and Dunwoody, forming a geographic panhandle. Major roads include Dunwoody Club Drive and Spalding Drive, and Interstate access is through the Northridge Road exit of Georgia 400. Many who lived in the neighborhood during Sandy Springs' incorporation considered themselves part of Dunwoody, and voiced their opposition to the installment of street sign toppers labelled "Sandy Springs". Then-mayor Eva Galambos stated that these new signs would do nothing to diminish the neighborhood's identity. Some residents still consider the area to be "Dunwoody in Sandy Springs", similar to the Buckhead Community district of Atlanta.

Perimeter Center is a commercial edge city and business district surrounding Perimeter Mall. Although about 40% of Perimeter Center, including the mall, is located in Dunwoody, the western 60%, including most of the area's office towers, are located in Sandy Springs. Pill Hill is located in the Sandy Springs section of Perimeter Center, and is the largest medical center in Georgia. It includes Northside Hospital, St. Joseph's Hospital, and Children's Healthcare of Atlanta. More than 40% of the hospital beds in the metro area are located within Sandy Springs. Landmarks include Hammond Park, Concourse at Landmark Center, colloquially called the 'King' and 'Queen' buildings due to their distinct white crown architecture at the top of each tower, as well as two MARTA stations, the Sandy Springs and Medical Center MARTA stations. The area also includes the 400-285 highway interchange, which is currently undergoing major construction. The top three tallest suburban buildings in the country are found here, the 'King' and 'Queen', and nearby Park Towers at #3.

North Springs is located in the northern portion of the city, and is generally defined as the area west of the Dunwoody/DeKalb County border, east of Brandon Mill Road, north of Abernathy Road, and south of Dalrymple Road and Spalding Drive. The North Springs MARTA station, the terminus of the MARTA Red Line, serves the district. Five of Sandy Springs' public schools are in this area, including the newest Ison Springs Elementary School, Woodland Elementary School, Spalding Drive Charter Elementary School, Sandy Springs Charter Middle School, and North Springs Charter High School.

South Springs or Sandy Springs ITP, an acronym for "inside the perimeter", refers to a portion of the city which extends south of Interstate 285, colloquially referred to as "the perimeter". It is located north of the City of Atlanta border, east of the Riverside district, south of Interstate 285, and west of the Brookhaven/DeKalb County border. The southern area of this district is considered to be a part of the greater Chastain Park community of Buckhead. The public schools in this area include Ridgeview Charter School and High Point Elementary School. The popular Atlanta radio station 99X broadcasts on 98.9 from here.

The business district just east of the river crossing is called Powers Ferry Landing, located where Northside Drive crosses the road, just east of the former landing. This provides freeway access at Northside Drive (west ramps) and New Northside Drive (east ramps, road and ramps built in a 1990s reconstruction). Signage on the freeway indicates Powers Ferry Road, Northside Drive, and New Northside Drive.

The North End is a large district in the northernmost portion of the city, and is generally defined as the area to the west of GA400, to the south and east of the Chattahoochee River, and to the north of Dalrymple Road. It is accessible via GA400 at Northridge Road, and contains the Northridge business area and the North River Village community. The Huntcliff community is located west of the district, on a panhandle to the northwest.

(Note: the 2000 U.S. census numbers are for Sandy Springs prior to incorporation, but cover the same area.)

Sandy Springs was first listed as a census designated place in the 1980 U.S. census and incorporated prior to the 2010 U.S. census.

According to the 2020 United States census, there were 108,080 people, 52,820 households, and 25,861 families residing in the city; this is up from a population of 93,853 at the 2010 census, and 85,781 at the 2000 census. When it was first listed as a census designated place in 1980, its population was 46,877.

According to a 2008 estimate, the median income for a household in the city was $106,240, and the median income for a family was $129,810. The average income for a household was $116,406 and the average income for a family was $169,815. Males had a median income of $60,053 versus $50,030 for females. About 3.1% of families and 7.9% of the population were below the poverty line, including 8.9% of those under age 18 and 1.9% of those age 65 or over.

• Mayor: Rusty Paul

Sandy Springs was noted for contracting private companies to perform the majority of its services in a public-private partnership model of government at the beginning of its incorporation in 2005. While many governments contract with private-sector companies on a per-project basis, Sandy Springs is believed to be the first American city to outsource its services for the majority of ongoing operations. They chose to do so as an economic response to the Great Recession. The city regularly hosted delegations from other governments that were interested in the model. Services not outsourced include police, fire-rescue, and city management. The city moved away from the private-public partnership model in 2019 when it was realized how much money was lost to private contractors and hired 184 full-time city staff that work at the new City Springs development. It now operates as a hybrid model, outsourcing projects to private companies as needed. The city estimates $14 million will be saved over the next five years from hiring full-time staff.

A new city hall opened in 2018.

Public schools are operated by the Fulton County School System. Elementary schools serving sections of Sandy Springs include Dunwoody Springs Charter Elementary School, Heards Ferry Elementary School, High Point Elementary School, Ison Springs Elementary School, Lake Forest Elementary School, Spalding Drive Charter Elementary School, and Woodland Charter Elementary School. Two middle schools, Sandy Springs Middle School and Ridgeview Charter Middle School, and two high schools, North Springs Charter School of Arts and Sciences and Riverwood High School, are in and serve Sandy Springs.

Private schools located in Sandy Springs include:

• Brandon Hall School (5th grade through high school)
• Springmont (formerly First Montessori School of Atlanta) (preschool through middle school)
• Atlanta Jewish Academy (K–12)
• Holy Innocents' Episcopal School (preschool through high school)
• Mount Vernon Presbyterian School (preschool through high school)
• St. Jude the Apostle Catholic School (K–8)
  • Opened September 4, 1962
• The Alfred and Adele Davis Academy (K–8)
• The Felicia Penzell Weber Jewish Community High School a.k.a. The Weber School (high school)
• The Epstein School (K–8)
• Holy Spirit Preparatory School Lower Campus (the upper campus and preschool are in Atlanta)
• Cumberland Academy

The initial campus of Sophia Academy, which opened in 1999, was on a rental property, in what became Sandy Springs. Construction on its new campus on what later became Chamblee began circa 2007.