Are you also going to build the infrastructure to support your apartment building in a neighborhood of single-family homes? Are you going widen the streets, create parking, improve sewer and water lines, add traffic lights, build larger parks, route mass transit, hire more school bus drivers, hire more police? Urban planning and zoning exist for a reason.
If I'm happy to have gains when I do nothing but my neighbours improve their properties, then I’m also happy to risk losses when I do nothing but my neighbours make changes.