Disruption of the neuronal membrane by toxic amyloid β oligomers is hypothesized to be the major event associated with Alzheimer's disease's neurotoxicity. Misfolding of amyloid β is followed by aggregation via different pathways in which structurally different amyloid β oligomers can be formed. The respective toxic actions of these structurally diverse oligomers can vary significantly. Linking a particular toxic action to a structurally unique kind of amyloid β oligomers and resolving their toxicity-determining feature remains challenging because of their transient stability and heterogeneity. Moreover, the lipids that make up the membrane affect amyloid β oligomers' behavior, thus adding to the problem's complexity. The present review compares and analyzes the latest results to improve understanding of amyloid β oligomers' interaction with lipid bilayers.