vmmon: check presence of file_operations::poll()

Commit 11c5ad0ec441 ("eventpoll: switch to ->poll_mask") in v4.18-rc1, made
eventpoll switch from ->poll() to ->poll_mask(), resulting in a null
pointer dereference as vmmon calls the callback directly without checking
its availability.

Rather than calling the callback directly (which would result in null
pointer dereference), use vfs_poll() wrapper. As this wrapper is only
available since 4.18-rc1 cycle as well, provide a copy to use when building
against older kernels.

Note: even if the commit revealing the problem was reverted in 4.18-rc3
(and is unlikely to go back in the same form) so that vmmon no longer
crashes with 4.18 kernel, calling the callback without a check is a bad
practice so let's keep the patch in place.

vmmon-only/COPYING

@@ -0,0 +1,339 @@
+		    GNU GENERAL PUBLIC LICENSE
+		       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+			    Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Lesser General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+		    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+			    NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+		     END OF TERMS AND CONDITIONS
+
+	    How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) year name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.

vmmon-only/Makefile

@@ -0,0 +1,147 @@
+#!/usr/bin/make -f
+##########################################################
+# Copyright (C) 1998-2019 VMware, Inc. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation version 2 and no later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+#
+##########################################################
+
+####
+####  VMware kernel module Makefile to be distributed externally
+####
+
+####
+#### SRCROOT _must_ be a relative path.
+####
+SRCROOT = .
+
+#
+# open-vm-tools doesn't replicate shared source files for different modules;
+# instead, files are kept in shared locations. So define a few useful macros
+# to be able to handle both cases cleanly.
+#
+INCLUDE      :=
+ifdef OVT_SOURCE_DIR
+AUTOCONF_DIR := $(OVT_SOURCE_DIR)/modules/linux/shared/autoconf
+VMLIB_PATH   = $(OVT_SOURCE_DIR)/lib/$(1)
+INCLUDE      += -I$(OVT_SOURCE_DIR)/modules/linux/shared
+INCLUDE      += -I$(OVT_SOURCE_DIR)/lib/include
+else
+AUTOCONF_DIR := $(SRCROOT)/shared/autoconf
+INCLUDE      += -I$(SRCROOT)/shared
+endif
+
+
+VM_UNAME = $(shell uname -r)
+
+# Header directory for the running kernel
+ifdef LINUXINCLUDE
+HEADER_DIR = $(LINUXINCLUDE)
+else
+HEADER_DIR = /lib/modules/$(VM_UNAME)/build/include
+endif
+
+BUILD_DIR = $(HEADER_DIR)/..
+
+DRIVER := vmmon
+PRODUCT := @@PRODUCT@@
+
+# Grep program
+GREP = /bin/grep
+
+vm_check_gcc = $(shell if $(CC) $(1) -S -o /dev/null -xc /dev/null \
+        > /dev/null 2>&1; then echo "$(1)"; else echo "$(2)"; fi)
+vm_check_file = $(shell if test -f $(1); then echo "yes"; else echo "no"; fi)
+
+ifndef VM_KBUILD
+VM_KBUILD := no
+ifeq ($(call vm_check_file,$(BUILD_DIR)/Makefile), yes)
+VM_KBUILD := yes
+endif
+export VM_KBUILD
+endif
+
+ifndef VM_KBUILD_SHOWN
+ifeq ($(VM_KBUILD), no)
+VM_DUMMY := $(shell echo >&2 "Using standalone build system.")
+else
+VM_DUMMY := $(shell echo >&2 "Using kernel build system.")
+endif
+VM_KBUILD_SHOWN := yes
+export VM_KBUILD_SHOWN
+endif
+
+ifneq ($(VM_KBUILD), no)
+
+# If there is no version defined, we are in toplevel pass, not yet in kernel makefiles...
+ifeq ($(VERSION),)
+
+DRIVER_KO := $(DRIVER).ko
+
+.PHONY: $(DRIVER_KO)
+
+auto-build: $(DRIVER_KO)
+	cp -f $< $(SRCROOT)/../$(DRIVER).o
+
+# $(DRIVER_KO) is a phony target, so compare file times explicitly
+$(DRIVER): $(DRIVER_KO)
+	if [ $< -nt $@ ] || [ ! -e $@ ] ; then cp -f $< $@; fi
+
+# Use SUBDIRS on 2.x, 3.x, 4.x.  Use M on newer kernels.
+ifeq ($(filter-out 2 3 4,$(firstword $(subst ., ,$(VM_UNAME)))),)
+DIRVAR := SUBDIRS
+else
+DIRVAR := M
+endif
+
+#
+# Define a setup target that gets built before the actual driver.
+# This target may not be used at all, but if it is then it will be defined
+# in Makefile.kernel
+#
+prebuild:: ;
+postbuild:: ;
+
+$(DRIVER_KO): prebuild
+	$(MAKE) -C $(BUILD_DIR) $(DIRVAR)=$$PWD SRCROOT=$$PWD/$(SRCROOT) \
+	  MODULEBUILDDIR=$(MODULEBUILDDIR) modules
+	$(MAKE) -C $$PWD SRCROOT=$$PWD/$(SRCROOT) \
+	  MODULEBUILDDIR=$(MODULEBUILDDIR) postbuild
+endif
+
+vm_check_build = $(shell if $(CC) $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) \
+	$(CPPFLAGS) $(CFLAGS) $(CFLAGS_KERNEL) $(LINUXINCLUDE) \
+	$(EXTRA_CFLAGS) -Iinclude2/asm/mach-default \
+	-DKBUILD_BASENAME=\"$(DRIVER)\" \
+	-Werror -S -o /dev/null -xc $(1) \
+	> /dev/null 2>&1; then echo "$(2)"; else echo "$(3)"; fi)
+
+CC_WARNINGS := -Wall -Wstrict-prototypes
+CC_OPTS := $(GLOBAL_DEFS) $(CC_WARNINGS) -DVMW_USING_KBUILD
+ifdef VMX86_DEVEL
+CC_OPTS += -DVMX86_DEVEL
+endif
+ifdef VMX86_DEBUG
+CC_OPTS += -DVMX86_DEBUG
+endif
+
+include $(SRCROOT)/Makefile.kernel
+
+else
+
+include $(SRCROOT)/Makefile.normal
+
+endif
+
+#.SILENT:

vmmon-only/Makefile.kernel

@@ -0,0 +1,37 @@
+#!/usr/bin/make -f
+##########################################################
+# Copyright (C) 1998,2015 VMware, Inc. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation version 2 and no later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+#
+##########################################################
+
+CC_OPTS += -DVMMON -DVMCORE
+
+INCLUDE := -I$(SRCROOT)/include -I$(SRCROOT)/common -I$(SRCROOT)/linux \
+	   -I$(SRCROOT)/vmcore
+
+EXTRA_CFLAGS := $(CC_OPTS) $(INCLUDE)
+
+obj-m += $(DRIVER).o
+
+$(DRIVER)-y := $(subst $(SRCROOT)/, , $(patsubst %.c, %.o, \
+		$(wildcard $(SRCROOT)/linux/*.c $(SRCROOT)/common/*.c $(SRCROOT)/vmcore/*.c \
+		$(SRCROOT)/bootstrap/*.c)))
+
+clean:
+	rm -rf $(wildcard $(DRIVER).mod.c $(DRIVER).ko .tmp_versions .cache.mk \
+		Module.symvers Modules.symvers Module.markers modules.order \
+		$(foreach dir,linux/ common/ vmcore/ bootstrap/ \
+		./,$(addprefix $(dir),.*.cmd .*.o.flags *.o)))

vmmon-only/Makefile.normal

@@ -0,0 +1,151 @@
+#!/usr/bin/make -f
+##########################################################
+# Copyright (C) 1998,2015,2017 VMware, Inc. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation version 2 and no later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+#
+##########################################################
+
+vm_check_build = $(shell if $(CC) $(CC_OPTS) $(INCLUDE) -Werror -S -o /dev/null -xc $(1) \
+        > /dev/null 2>&1; then echo "$(2)"; else echo "$(3)"; fi)
+
+####
+####  DESTDIR is where the module, object files, and dependencies are built
+####
+DESTDIR := driver-$(VM_UNAME)
+
+####
+####  DRIVERNAME should be untouched unless you have a good reason to change
+#### it.  The form below is how the scripts expect it.
+####
+DRIVERNAME := $(DRIVER)-xxx-$(VM_UNAME)
+
+ifneq (,$(filter x86_64%, $(shell $(CC) -dumpmachine)))
+MACHINE := x86_64
+else
+MACHINE := x386
+endif
+
+ifdef QUIET
+ECHO := @true
+else
+ECHO := @echo
+endif
+
+####
+#### You must compile with at least -O level of optimization
+#### or the module won't load.
+#### If desparate, I think that bringing in <linux/bitops.h> might
+#### suffice.
+####
+CC_WARNINGS := -Wall -Wstrict-prototypes
+# Don't use -pipe or egcs-2.91.66 (shipped with RedHat) will die
+CC_KFLAGS := -D__KERNEL__ -fno-strength-reduce -fno-omit-frame-pointer \
+	     -fno-common -DKBUILD_MODNAME=$(DRIVER)
+CC_KFLAGS += $(call vm_check_gcc,-falign-loops=2 -falign-jumps=2 -falign-functions=2, \
+	     -malign-loops=2 -malign-jumps=2 -malign-functions=2)
+CC_KFLAGS += $(call vm_check_gcc,-fno-strict-aliasing,)
+ifeq ($(MACHINE),x86_64)
+CC_KFLAGS += -mno-red-zone -mcmodel=kernel
+else
+# Gcc 3.0 deprecates -m486 --hpreg
+CC_KFLAGS += -DCPU=586 $(call check_gcc,-march=i586,-m486)
+endif
+
+CC_OPTS := -O2 -DMODULE -DVMMON -DVMCORE $(GLOBAL_DEFS) $(CC_KFLAGS) $(CC_WARNINGS)
+
+INCLUDE := -I$(SRCROOT)/include -I$(SRCROOT)/common -I$(SRCROOT)/linux \
+	   -I$(SRCROOT)/vmcore -I$(HEADER_DIR)
+
+INCLUDE += $(shell $(CC) $(INCLUDE) -E $(SRCROOT)/autoconf/geninclude.c \
+	   | sed -n -e 's!^APATH!-I$(HEADER_DIR)/asm!p')
+
+C_TARGETS_LINUX  := driver.o hostif.o driverLog.o
+C_TARGETS_COMMON := vmx86.o memtrack.o phystrack.o cpuid.o task.o
+C_TARGETS_VMCORE := moduleloop.o
+C_TARGETS_BOOTSTRAP := bootstrap.o
+C_TARGETS_LINUX_D := ${C_TARGETS_LINUX:.o=.d}
+C_TARGETS_COMMON_D := ${C_TARGETS_COMMON:.o=.d}
+C_TARGETS_VMCORE_D := ${C_TARGETS_VMCORE:.o=.d}
+C_TARGETS_BOOTSTRAP_D := ${C_TARGETS_BOOTSTRAP:.o=.d}
+C_TARGETS := $(C_TARGETS_LINUX) $(C_TARGETS_COMMON) $(C_TARGETS_VMCORE) \
+             $(C_TARGETS_BOOTSTRAP)
+
+####
+#### Make Targets are beneath here.
+####
+
+driver: setup deps
+	$(MAKE) -C $(DESTDIR) -f ../Makefile SRCROOT=../$(SRCROOT) $(DRIVER).o \
+		INCLUDE_DEPS=1
+
+setup:
+	@if [ -d $(DESTDIR) ] ; then true ; else mkdir $(DESTDIR); chmod 755 $(DESTDIR) ; fi
+
+$(DRIVER) $(DRIVER).o: $(DRIVERNAME)
+	cp -f $< $@
+
+$(DRIVERNAME): $(C_TARGETS)
+	$(ECHO) "Building $(DRIVERNAME)"
+	ld -r -o $(DRIVERNAME) $(C_TARGETS)
+
+auto-build:
+	$(MAKE) driver QUIET=1
+	cp -f $(DESTDIR)/$(DRIVERNAME) $(SRCROOT)/../$(DRIVER).o
+
+$(C_TARGETS_LINUX): %.o: $(SRCROOT)/linux/%.c
+	$(ECHO) "Compiling  linux/$(<F)"
+	$(CC) $(CC_OPTS) $(INCLUDE) -c $<
+
+$(C_TARGETS_COMMON): %.o: $(SRCROOT)/common/%.c
+	$(ECHO) "Compiling  common/$(<F)"
+	$(CC) $(CC_OPTS) $(INCLUDE) -c $<
+
+$(C_TARGETS_VMCORE): %.o: $(SRCROOT)/vmcore/%.c
+	$(ECHO) "Compiling  vmcore/$(<F)"
+	$(CC) $(CC_OPTS) $(INCLUDE) -c $<
+
+$(C_TARGETS_BOOTSTRAP): %.o: $(SRCROOT)/bootstrap/%.c
+	$(ECHO) "Compiling  bootstrap/$(<F)"
+	$(CC) $(CC_OPTS) $(INCLUDE) -c $<
+
+clean:
+	rm -rf $(DESTDIR)/
+
+$(C_TARGETS_COMMON_D): %.d: $(SRCROOT)/common/%.c
+	$(ECHO) "Dependencies for $(<F)"
+	$(CC) -MM $(CC_OPTS) $(INCLUDE) $< > $@
+
+$(C_TARGETS_LINUX_D): %.d: $(SRCROOT)/linux/%.c
+	$(ECHO) "Dependencies for $(<F)"
+	$(CC) -MM $(CC_OPTS) $(INCLUDE) $< > $@
+
+$(C_TARGETS_VMCORE_D): %.d: $(SRCROOT)/vmcore/%.c
+	$(ECHO) "Dependencies for $(<F)"
+	$(CC) -MM $(CC_OPTS) $(INCLUDE) $< > $@
+
+$(C_TARGETS_BOOTSTRAP_D): %.d: $(SRCROOT)/bootstrap/%.c
+	$(ECHO) "Dependencies for $(<F)"
+	$(CC) -MM $(CC_OPTS) $(INCLUDE) $< > $@
+
+deps: setup
+	$(MAKE) -C $(DESTDIR) -f ../Makefile SRCROOT=../$(SRCROOT) driver_deps
+
+driver_deps: ${C_TARGETS:.o=.d}
+
+ifdef INCLUDE_DEPS
+include ${C_TARGETS:.o=.d}
+endif
+
+.SILENT:

vmmon-only/README

@@ -0,0 +1,14 @@
+This files in this directory and its subdirectories are the kernel module
+portion of the VMware Virtual Machine Monitor.  In order to build, make
+certain the Makefile is correct, especially in whether or not your system
+is multi-processor and then just type
+
+        make
+
+from this directory.  A copy of the module will be left in
+
+        driver-<kernel version>/vmmon.o
+
+(e.g. driver-up-2.0.32/vmmon.o).
+
+If you have any problems or questions, send mail to support@vmware.com

vmmon-only/autoconf/geninclude.c

@@ -0,0 +1,41 @@
+/*********************************************************
+ * Copyright (C) 2003 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#include "compat_version.h"
+#include "compat_autoconf.h"
+
+#ifdef CONFIG_X86_VOYAGER
+APATH/mach-voyager
+#endif
+#ifdef CONFIG_X86_VISWS
+APATH/mach-visws
+#endif
+#ifdef CONFIG_X86_NUMAQ
+APATH/mach-numaq
+#endif
+#ifdef CONFIG_X86_BIGSMP
+APATH/mach-bigsmp
+#endif
+#ifdef CONFIG_X86_SUMMIT
+APATH/mach-summit
+#endif
+#ifdef CONFIG_X86_GENERICARCH
+APATH/mach-generic
+#endif
+APATH/mach-default
+

vmmon-only/bootstrap/bootstrap.c

@@ -0,0 +1,53 @@
+/*********************************************************
+ * Copyright (C) 2015 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * bootstrap.c --
+ *
+ *    Implements the early VMM bootstraping code that is executed
+ *    by the host (vmmon/VMKernel) to create the VMM context.
+ */
+
+#include "vm_basic_types.h"
+#include "vm_basic_defs.h"
+#include "bootstrap_vmm.h"
+
+/*
+ *---------------------------------------------------------------------
+ *
+ * BSVMM_Validate --
+ *
+ *    Validates the VMM bootstrap blob. For now, we do it by checking
+ *    the magic number. Returns the bootstrap parameter table if
+ *    successful, NULL otherwise.
+ *
+ *---------------------------------------------------------------------
+ */
+BSVMM_HostParams *
+BSVMM_Validate(void *buf, uint32 nbytes)
+{
+   BSVMM_HostParams *bsParams = buf;
+
+   if (nbytes < sizeof *bsParams) {
+      return NULL;
+   }
+   if (bsParams->magic != BOOTSTRAP_MAGIC) {
+      return NULL;
+   }
+   return bsParams;
+}

vmmon-only/bootstrap/monLoader.c

@@ -0,0 +1,963 @@
+/*********************************************************
+ * Copyright (C) 2015-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * monLoader.c --
+ *
+ *      Processes the monitor loader header.
+ *
+ *      See monLoader.h for a full description.
+ */
+
+#if defined VMKERNEL
+
+#include "vmkernel.h"
+#include "libc.h"
+#include "vm_libc.h"
+
+#define UINT_MAX (~0U)
+
+#elif defined VMMON
+
+#include "hostKernel.h"
+
+#ifdef __linux__
+#   include "driver-config.h"
+#   include <linux/string.h>
+#   include <linux/kernel.h>
+#elif defined __APPLE__
+#   include <machine/limits.h>
+#   include <string.h>
+#else
+#   include <limits.h>
+#   include <string.h>
+#endif
+
+#include "vm_assert.h"
+
+#else /* !defined VMKERNEL && !defined VMMON */
+#error MonLoader cannot be built as part of this environment
+#endif
+
+#include "vm_basic_types.h"
+#include "monLoader.h"
+#include "vcpuid.h"
+#include "vm_pagetable.h"
+#include "address_defs.h"
+#include "monLoaderLog.h"
+
+#define CANONICAL_MASK MASK64(36)
+/* The index of VPN v within an array of LxEs where index 0 maps VPN b. */
+#define L4EArrayIdx(b,v) ((unsigned)((((v) - (b)) & CANONICAL_MASK) >> 36))
+#define L3EArrayIdx(b,v) ((unsigned)((((v) - (b)) & CANONICAL_MASK) >> 27))
+#define L2EArrayIdx(b,v) ((unsigned)((((v) - (b)) & CANONICAL_MASK) >> 18))
+#define L1EArrayIdx(b,v) ((unsigned)((((v) - (b)) & CANONICAL_MASK) >> 9))
+
+/* The maximum (canonical-address) VPN */
+#define VPN_MAX MASK64(52)
+
+/* Sufficient MPN counts to accommodate the monitor's top 64MB. */
+#define L4MPNCOUNTMAX  1
+#define L3MPNCOUNTMAX  1
+#define L2MPNCOUNTMAX  1
+#define L1MPNCOUNTMAX 32
+
+#define LINE_INVALID  ((unsigned)-1)
+
+typedef struct MonPTMPNs {
+   MPN L4MPNs[L4MPNCOUNTMAX]; /* The page root.  Definitely only one page. */
+   MPN L3MPNs[L3MPNCOUNTMAX];
+   MPN L2MPNs[L2MPNCOUNTMAX];
+   MPN L1MPNs[L1MPNCOUNTMAX];
+   unsigned L4MPNCount;
+   unsigned L3MPNCount;
+   unsigned L2MPNCount;
+   unsigned L1MPNCount;
+} MonPTMPNs;
+ 
+typedef struct MonLoaderContext {
+   struct MonLoaderEnvContext *envCtx; /* Environment-specific context */
+   struct {
+      MonPTMPNs    ptMPNs;      /* Mappings into the AS for the current VCPU. */
+      VPN          ASFirstVPN;  /* first VPN in the address space */
+      VPN          ASLastVPN;   /* last VPN in the address space (inclusive) */
+      uint64       ASPTEFlags;  /* PTE flags for L4->L1 connection. */
+      Vcpuid       currentVCPU;
+      Bool         hasAddrSpace;
+   } vcpu;
+} MonLoaderContext;
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderTranslateMonVPNToL1E --
+ *
+ *      Translates a monitor VPN to a level 1 page table entry, for the current
+ *      VCPU.  Assumes preallocated and connected page tables, so only the L1
+ *      table is examined.
+ *
+ * Result:
+ *      ML_OK and a PTE if successful, some other value on error.  Fails if a
+ *      page table page is not already allocated or the VPN is not in the
+ *      context's address space's defined range.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderTranslateMonVPNToL1E(MonLoaderContext *ctx,    // IN/OUT
+                              VPN               monVPN, // IN
+                              PT_L1E           *pte)    // OUT
+{
+   Vcpuid vcpu = ctx->vcpu.currentVCPU;
+   /* Only modify the L1E as L4->L1 are guaranteed connected already. */
+   unsigned L1Off = PT_LPN_2_L1OFF(monVPN);
+   unsigned L1Page = L1EArrayIdx(ctx->vcpu.ASFirstVPN, monVPN);
+
+   if (monVPN < ctx->vcpu.ASFirstVPN || monVPN > ctx->vcpu.ASLastVPN) {
+      return ML_ERROR_INVALID_VPN; /* Address is outside address space. */
+   }
+   if (ctx->vcpu.ptMPNs.L1MPNs[L1Page] == INVALID_MPN) {
+      return ML_ERROR_MAP; /* Page table not sufficiently preallocated. */
+   }
+   if (!MonLoaderCallout_GetPTE(ctx->envCtx, ctx->vcpu.ptMPNs.L1MPNs[L1Page],
+                                L1Off, vcpu, pte)) {
+      return ML_ERROR_CALLOUT_GETPTE;
+   }
+   return ML_OK;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderIsMapped --
+ *
+ *      Checks whether a VPN has an existing mapping on the current VCPU.
+ *      Assumes preallocated and connected page tables.
+ *
+ * Result:
+ *      ML_OK if successful, some other value on error.  Fails if a page table
+ *      page is not already allocated or the VPN is not in the context's
+ *      address space's defined range.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderIsMapped(MonLoaderContext *ctx,    // IN/OUT
+                  VPN               monVPN, // IN
+                  Bool             *mapped) // OUT
+{
+   PT_L1E pte;
+   MonLoaderError ret = MonLoaderTranslateMonVPNToL1E(ctx, monVPN, &pte);
+
+   if (ret != ML_OK) {
+      return ret;
+   }
+   *mapped = ML_PERM_PRESENT(pte);
+   return ML_OK;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderAllocMPN --
+ *
+ *      Allocates a new MPN.
+ *
+ * Result:
+ *      ML_OK and an MPN, on success.  An error and no MPN on failure.
+ *
+ * Side effects:
+ *      An MPN may be allocated.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderAllocMPN(MonLoaderContext *ctx, // IN
+                  MPN *mpn)              // OUT
+{
+   *mpn = MonLoaderCallout_AllocMPN(ctx->envCtx, ctx->vcpu.currentVCPU);
+   if (*mpn == INVALID_MPN) {
+      return ML_ERROR_ALLOC;
+   }
+   return ML_OK;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderMapMPN --
+ *
+ *      Maps an MPN into the current VCPU's address space at a VPN.
+ *
+ * Result:
+ *      ML_OK if successful, some other value on error.  Fails if a page table
+ *      page is not already allocated or the VPN is not in the context's
+ *      address space's defined range.
+ *
+ * Side effects:
+ *      An MPN is mapped on success.  Nothing on failure.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderMapMPN(MonLoaderContext *ctx,    // IN/OUT
+                MPN               mpn,    // IN
+                uint64            flags,  // IN
+                VPN               monVPN) // IN
+{
+   MonPTMPNs *ptMPNs = &ctx->vcpu.ptMPNs;
+   /* Only modify the L1E, as L4->L1 are guaranteed connected already. */
+   unsigned L1Off = PT_LPN_2_L1OFF(monVPN);
+   unsigned L1Page = L1EArrayIdx(ctx->vcpu.ASFirstVPN, monVPN);
+
+   if (vmx86_debug) {
+      MonLoaderError ret;
+      Bool mapped;
+      if (monVPN < ctx->vcpu.ASFirstVPN || monVPN > ctx->vcpu.ASLastVPN) {
+         LOG(5, "%s: monVPN=0x%"FMTVPN"x, AS=0x%"FMTVPN"x-0x%"FMTVPN"x\n",
+                 __FUNCTION__, monVPN, ctx->vcpu.ASFirstVPN,
+                 ctx->vcpu.ASLastVPN);
+         return ML_ERROR_INVALID_VPN; /* Address is outside address space. */
+      }
+      if (ptMPNs->L1MPNs[L1Page] == INVALID_MPN) {
+         return ML_ERROR_MAP; /* Page table not sufficiently preallocated. */
+      }
+      ret = MonLoaderIsMapped(ctx, monVPN, &mapped);
+      if (ret != ML_OK) {
+         return ret;
+      }
+      if (mapped) {
+         return ML_ERROR_ALREADY_MAPPED;
+      }
+   }
+   if (!MonLoaderCallout_MapMPNInPTE(ctx->envCtx, ptMPNs->L1MPNs[L1Page], L1Off,
+                                     flags, mpn, ctx->vcpu.currentVCPU)) {
+      return ML_ERROR_CALLOUT_MAPINPTE;
+   }
+   return ML_OK;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderBuildsPtLevel --
+ *
+ *      Determines whether MonLoader allocates and maps the page table(s) for
+ *      the monitor at the given level.
+ *
+ * Result:
+ *      TRUE if MonLoader creates the page table at the given level, FALSE
+ *      otherwise.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+static Bool
+MonLoaderCreatesPtLevel(PT_Level level)
+{
+   ASSERT(level >= PT_LEVEL_STOP && level <= PT_MAX_LEVELS);
+   return FALSE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCreateAddressSpace --
+ *
+ *      Creates or verifies an address space.  The VPN range specified by
+ *      firstVPN and size is used to determine page counts at each page table
+ *      level to map every page.  Sufficient pages are then allocated or
+ *      verified.
+ *
+ * Result:
+ *      ML_OK if successful, some other value on error.
+ *
+ * Side effects:
+ *      MPNs are allocated if not importing.  Sets ctx->vcpu.hasAddrSpace.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderCreateAddressSpace(MonLoaderContext *ctx,      // IN/OUT
+                            VPN               firstVPN, // IN
+                            uint64            flags,    // IN
+                            uint64            monPages) // IN
+{
+   Vcpuid vcpu = ctx->vcpu.currentVCPU;
+   VPN lastVPN = firstVPN + monPages - 1;
+   unsigned L3MPNsNeeded = L3EArrayIdx(0, lastVPN) -
+                           L3EArrayIdx(0, firstVPN) + 1;
+   unsigned L2MPNsNeeded = L2EArrayIdx(0, lastVPN) -
+                           L2EArrayIdx(0, firstVPN) + 1;
+   unsigned L1MPNsNeeded = L1EArrayIdx(0, lastVPN) -
+                           L1EArrayIdx(0, firstVPN) + 1;
+   unsigned i;
+   MPN mpn;
+   MonPTMPNs *ptMPNs;
+
+   if (vcpu == VCPUID_INVALID) {
+      return ML_ERROR_ARGS;
+   }
+
+   /* Verify MonPTMPNs is large enough for this address space. */
+   if (L3MPNsNeeded > L3MPNCOUNTMAX || L2MPNsNeeded > L2MPNCOUNTMAX ||
+       L1MPNsNeeded > L1MPNCOUNTMAX) {
+      return ML_ERROR_ADDRSPACE_TOO_LARGE;
+   }
+   ctx->vcpu.ASFirstVPN = firstVPN;
+   ctx->vcpu.ASLastVPN = lastVPN;
+   ctx->vcpu.ASPTEFlags = flags;
+
+   ptMPNs = &ctx->vcpu.ptMPNs;
+
+   if (MonLoaderCreatesPtLevel(PT_LEVEL_4)) {
+      NOT_IMPLEMENTED();
+   } else {
+      /* Verify the VMX's allocation. */
+      mpn = MonLoaderCallout_GetPageRoot(ctx->envCtx, vcpu);
+      if (mpn == INVALID_MPN) {
+         return ML_ERROR_CALLOUT_PAGEROOT_GET;
+      }
+      LOG(5, "%s: vcpu %u page root=0x%"FMT64"x\n", __FUNCTION__, vcpu, mpn);
+      ptMPNs->L4MPNs[0] = mpn;
+      MonLoaderCallout_ImportPage(ctx->envCtx, mpn, vcpu);
+   }
+   ptMPNs->L4MPNCount = 1;
+
+   for (i = 0; i < L3MPNsNeeded; i++) {
+      if (MonLoaderCreatesPtLevel(PT_LEVEL_3)) {
+         NOT_IMPLEMENTED();
+      } else {
+         VPN monVPN = firstVPN + i * PT_PAGES_PER_L4E;
+         unsigned L4Off = PT_LPN_2_L4OFF(monVPN);
+         PT_L1E pte;
+         if (!MonLoaderCallout_GetPTE(ctx->envCtx, ptMPNs->L4MPNs[0], L4Off,
+                                      vcpu, &pte)) {
+            return ML_ERROR_CALLOUT_GETPTE;
+         }
+         LOG(5, "%s: monVPN=0x%"FMTVPN"x: L4E=0x%"FMT64"x\n", __FUNCTION__,
+                 monVPN, pte);
+         mpn = ML_PTE_2_PFN(pte);
+         if (mpn == INVALID_MPN || !ML_PERMS_MATCH(pte, flags)) {
+            return ML_ERROR_PAGE_TABLE_IMPORT;
+         }
+         MonLoaderCallout_ImportPage(ctx->envCtx, mpn, vcpu);
+      }
+      ptMPNs->L3MPNs[i] = mpn;
+   }
+   ptMPNs->L3MPNCount = L3MPNsNeeded;
+      
+   for (i = 0; i < L2MPNsNeeded; i++) {
+      if (MonLoaderCreatesPtLevel(PT_LEVEL_2)) {
+         NOT_IMPLEMENTED();
+      } else {
+         VPN monVPN = firstVPN + i * PT_PAGES_PER_L3E;
+         unsigned L3Off = PT_LPN_2_L3OFF(monVPN);
+         unsigned L3Page = L3EArrayIdx(ctx->vcpu.ASFirstVPN, monVPN);
+         PT_L1E pte;
+         if (!MonLoaderCallout_GetPTE(ctx->envCtx, ptMPNs->L3MPNs[L3Page],
+                                      L3Off, vcpu, &pte)) {
+            return ML_ERROR_CALLOUT_GETPTE;
+         }
+         LOG(5, "%s: monVPN=0x%"FMTVPN"x: L3E=0x%"FMT64"x\n", __FUNCTION__,
+                 monVPN, pte);
+         mpn = ML_PTE_2_PFN(pte);
+         if (mpn == INVALID_MPN || !ML_PERMS_MATCH(pte, flags)) {
+            return ML_ERROR_PAGE_TABLE_IMPORT;
+         }
+         MonLoaderCallout_ImportPage(ctx->envCtx, mpn, vcpu);
+      }
+      ptMPNs->L2MPNs[i] = mpn;
+   }
+   ptMPNs->L2MPNCount = L2MPNsNeeded;
+
+   for (i = 0; i < L1MPNsNeeded; i++) {
+      if (MonLoaderCreatesPtLevel(PT_LEVEL_1)) {
+         NOT_IMPLEMENTED();
+      } else {
+         VPN monVPN = firstVPN + i * PT_PAGES_PER_L2E;
+         unsigned L2Off = PT_LPN_2_L2OFF(monVPN);
+         unsigned L2Page = L2EArrayIdx(ctx->vcpu.ASFirstVPN, monVPN);
+         PT_L1E pte;
+         if (!MonLoaderCallout_GetPTE(ctx->envCtx, ptMPNs->L2MPNs[L2Page],
+                                      L2Off, vcpu, &pte)) {
+            return ML_ERROR_CALLOUT_GETPTE;
+         }
+         LOG(5, "%s: monVPN=0x%"FMTVPN"x: L2E=0x%"FMT64"x\n", __FUNCTION__,
+                 monVPN, pte);
+         if (!ML_PERM_PRESENT(pte)) {
+            ptMPNs->L1MPNs[i] = INVALID_MPN;
+            continue;
+         }
+         mpn = ML_PTE_2_PFN(pte);
+         if (mpn == INVALID_MPN || !ML_PERMS_MATCH(pte, flags)) {
+            return ML_ERROR_PAGE_TABLE_IMPORT;
+         }
+         MonLoaderCallout_ImportPage(ctx->envCtx, mpn, vcpu);
+      }
+      ptMPNs->L1MPNs[i] = mpn;
+   }
+   ptMPNs->L1MPNCount = L1MPNsNeeded;
+
+   ctx->vcpu.hasAddrSpace = TRUE;
+   return ML_OK;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderMapPageTables --
+ *
+ *      Maps in the page tables for the specified level, or verifies that the
+ *      existing mappings match internal state and flags if the address space
+ *      was imported.  Assumes pre-allocated memory at all levels and
+ *      pre-connected L4->L1 page tables.
+ *
+ * Result:
+ *      ML_OK if successful, some other value on error.
+ *
+ * Side effects:
+ *      Page tables are updated if mappings are created.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderMapPageTables(MonLoaderContext *ctx,      // IN/OUT
+                       PT_Level          level,    // IN
+                       uint64            flags,    // IN
+                       VPN               monVPN,   // IN
+                       uint64            monPages) // IN
+{
+   Bool verify = !MonLoaderCreatesPtLevel(level);
+   uint64    i;
+   unsigned  count;
+   MPN      *ptMPNs;
+
+   if (vmx86_debug) {
+      if (!ctx->vcpu.hasAddrSpace) {
+         return ML_ERROR_NO_ADDRSPACE;
+      }
+      if (level < PT_LEVEL_STOP || level > PT_MAX_LEVELS) {
+         return ML_ERROR_ARGS;
+      }
+   }
+
+   switch (level) {
+      case PT_LEVEL_4:
+         count = ctx->vcpu.ptMPNs.L4MPNCount;
+         ptMPNs = ctx->vcpu.ptMPNs.L4MPNs;
+         break;
+      case PT_LEVEL_3:
+         count = ctx->vcpu.ptMPNs.L3MPNCount;
+         ptMPNs = ctx->vcpu.ptMPNs.L3MPNs;
+         break;
+      case PT_LEVEL_2:
+         count = ctx->vcpu.ptMPNs.L2MPNCount;
+         ptMPNs = ctx->vcpu.ptMPNs.L2MPNs;
+         break;
+      case PT_LEVEL_1:
+         count = ctx->vcpu.ptMPNs.L1MPNCount;
+         ptMPNs = ctx->vcpu.ptMPNs.L1MPNs;
+         break;
+      default:
+         return ML_ERROR_ARGS;
+         break;
+
+      if (monPages != count) {
+         return ML_ERROR_PAGE_TABLE_MAP_SIZE;
+      }
+      
+      for (i = 0; i < monPages; i++) {
+         VPN vpn = monVPN + i;
+         if (verify) {
+            PT_L1E l1e;
+            if (ptMPNs[i] == INVALID_MPN && level == PT_LEVEL_1) {
+               continue;
+            }
+            if (MonLoaderTranslateMonVPNToL1E(ctx, vpn, &l1e) != ML_OK ||
+                ML_PTE_2_PFN(l1e) != ptMPNs[i] || !ML_PERMS_MATCH(l1e, flags)) {
+               return ML_ERROR_PAGE_TABLE_VERIFY;
+            }
+         } else {
+            NOT_IMPLEMENTED();
+         }
+      }
+   }
+   return ML_OK;
+} 
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderZero --
+ *
+ *      Allocates new pages, zeroes them and maps them on the current VCPU.
+ *
+ * Result:
+ *      ML_OK if successful, some other value on error.  The number of
+ *      MPNs allocated in *allocs is incremented.
+ *
+ * Side effects:
+ *      Allocations, zeroing, mapping and accounting are performed.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderZero(MonLoaderContext *ctx,      // IN/OUT
+              uint64            flags,    // IN
+              VPN               monVPN,   // IN
+              uint64            numPages, // IN
+              uint64            monPages, // IN
+              unsigned         *allocs)   // IN/OUT
+{
+   uint64 i;
+
+   if (numPages == 0 || numPages > monPages) {
+      return ML_ERROR_SIZE;
+   }
+   for (i = 0; i < numPages; i++) {
+      MPN mpn;
+      MonLoaderError ret = MonLoaderAllocMPN(ctx, &mpn);
+      if (ret != ML_OK) {
+         return ret;
+      }
+      if (!MonLoaderCallout_FillPage(ctx->envCtx, 0, mpn,
+                                     ctx->vcpu.currentVCPU)) {
+         return ML_ERROR_CALLOUT_ZERO;
+      }
+      ret = MonLoaderMapMPN(ctx, mpn, flags, monVPN + i);
+      if (ret != ML_OK) {
+         return ret;
+      }
+      *allocs += 1;
+   }
+   return ML_OK;
+}
+   
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCopyFromBlob --
+ *
+ *      Allocates new pages, copies blob contents and maps the pages.
+ *
+ * Result:
+ *      ML_OK if successful, some other value on error.  The number of
+ *      MPNs allocated in *allocs is incremented.
+ *
+ * Side effects:
+ *      Allocations, copying, mapping and accounting are performed.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderCopyFromBlob(MonLoaderContext *ctx,        // IN/OUT
+                      uint64            flags,      // IN
+                      VPN               monVPN,     // IN
+                      uint64            monBytes,   // IN
+                      uint64            blobOffset, // IN
+                      uint64            blobSize,   // IN
+                      unsigned         *allocs)     // IN/OUT
+{
+   uint64 bytesLeft = blobSize;
+
+   if (blobSize > monBytes || blobSize == 0) {
+      return ML_ERROR_SIZE; /* Entry corrupt: size mismatch. */
+   }
+   while (bytesLeft) {
+      MPN mpn;
+      uint64 toCopy = MIN(bytesLeft, PAGE_SIZE);
+
+      MonLoaderError ret = MonLoaderAllocMPN(ctx, &mpn);
+      if (ret != ML_OK) {
+         return ret;
+      }
+
+      ret = MonLoaderMapMPN(ctx, mpn, flags, monVPN);
+      if (ret != ML_OK) {
+         return ret;
+      }
+
+      if (!MonLoaderCallout_CopyFromBlob(ctx->envCtx, blobOffset, toCopy, mpn,
+                                         ctx->vcpu.currentVCPU)) {
+         return ML_ERROR_CALLOUT_COPY;
+      }
+
+      monVPN++;
+      blobOffset += toCopy;
+      bytesLeft -= toCopy;
+      *allocs += 1;
+   }
+   return ML_OK;
+}
+ 
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderShareWork --
+ *
+ *      Maps user or host pages in.  monPages is a maximum, so partial sharing
+ *      is considered successful and remaining monitor pages will not be mapped.
+ *
+ * Result:
+ *      ML_OK if successful, some other value on error.
+ *
+ * Side effects:
+ *      User or host pages may be mapped in.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderShareWork(MonLoaderContext *ctx,        // IN/OUT
+                   uint64            flags,      // IN
+                   VPN               monVPN,     // IN
+                   uint64            monPages,   // IN
+                   uint64            subIndex,   // IN
+                   Bool              user)       // IN
+{
+   Vcpuid vcpu = ctx->vcpu.currentVCPU;
+   uint64 i;
+
+   if (monPages == 0 || monPages > UINT_MAX) {
+      return ML_ERROR_SIZE; /* Entry corrupt: size mismatch. */
+   }
+   for (i = 0; i < monPages; i++) {
+      MPN mpn;
+      MonLoaderError ret;
+      unsigned pgNum = (unsigned)i;
+
+      if (user) {
+         mpn = MonLoaderCallout_GetSharedUserPage(ctx->envCtx, subIndex, pgNum,
+                                                  vcpu);
+      } else {
+         mpn = MonLoaderCallout_GetSharedHostPage(ctx->envCtx, subIndex, pgNum,
+                                                  vcpu);
+      }
+      if (mpn == INVALID_MPN) {
+         if (subIndex == MONLOADER_HT_MAP_IDX) {
+            /*
+             * This item is tied to a vmkernel feature.
+             * When the feature is disabled, there is nothing to share.
+             */
+            ASSERT(vmx86_server);
+            return ML_OK;
+         }
+         /* Partial sharing is allowed.  Return success if any occurred. */
+         return i != 0 ? ML_OK : ML_ERROR_SHARE;
+      }
+      ret = MonLoaderMapMPN(ctx, mpn, flags, monVPN + i);
+      if (ret != ML_OK) {
+         return ret;
+      }
+   }
+   return ML_OK;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderShareFromUser --
+ *
+ *      Maps user pages in.  monPages is a maximum, so partial sharing is
+ *      considered successful and remaining monitor pages will not be mapped.
+ *
+ * Result:
+ *      ML_OK if successful, some other value on error.
+ *
+ * Side effects:
+ *      User pages may be mapped in.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderShareFromUser(MonLoaderContext *ctx,        // IN/OUT
+                       uint64            flags,      // IN
+                       VPN               monVPN,     // IN
+                       uint64            monPages,   // IN
+                       uint64            subIndex)   // IN
+{
+   return MonLoaderShareWork(ctx, flags, monVPN, monPages, subIndex, TRUE);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderShareFromHost --
+ *
+ *      Maps host pages in.  monPages is a maximum, so partial sharing is
+ *      considered successful and remaining monitor pages will not be mapped.
+ *
+ * Result:
+ *      ML_OK if successful, some other value on error.
+ *
+ * Side effects:
+ *      Host pages may be mapped in.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderShareFromHost(MonLoaderContext *ctx,        // IN/OUT
+                       uint64            flags,      // IN
+                       VPN               monVPN,     // IN
+                       uint64            monPages,   // IN
+                       uint64            subIndex)   // IN
+{
+   return MonLoaderShareWork(ctx, flags, monVPN, monPages, subIndex, FALSE);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderShareFromBlob --
+ *
+ *      Maps blob pages in. monPages is a maximum, so partial sharing is
+ *      considered successful and remaining monitor pages will not be
+ *      mapped. blobOffset must be page-aligned and blobSize must be a
+ *      multiple of PAGE_SIZE. Writeable mappings are not allowed.
+ *
+ *     Result:
+ *          ML_OK if successful, some other value on error.
+ *
+ *     Side effects:
+ *          Blob pages may be mapped in.
+ *
+ *----------------------------------------------------------------------
+ */
+static MonLoaderError
+MonLoaderShareFromBlob(MonLoaderContext *ctx,        // IN/OUT
+                       uint64            flags,      // IN
+                       VPN               monVPN,     // IN
+                       uint64            monBytes,   // IN
+                       uint64            blobOffset, // IN
+                       uint64            blobSize)   // IN
+{
+   uint64 endOff = blobOffset + blobSize;
+
+   if ((blobOffset & (PAGE_SIZE - 1)) != 0 ||
+       (blobSize & (PAGE_SIZE - 1)) != 0 ||
+       ML_PERM_WRITEABLE(flags)) {
+      return ML_ERROR_SHARE;
+   }
+   if (blobSize > monBytes || blobSize == 0) {
+      return ML_ERROR_SIZE;
+   }
+   while (blobOffset < endOff) {
+      MonLoaderError ret;
+      MPN mpn = MonLoaderCallout_GetBlobMpn(ctx->envCtx, blobOffset);
+      if (mpn == INVALID_MPN) {
+         return ML_ERROR_INVALID_VPN;
+      }
+      ret = MonLoaderMapMPN(ctx, mpn, flags, monVPN);
+      if (ret != ML_OK) {
+         return ret;
+      }
+      blobOffset += PAGE_SIZE;
+      monVPN++;
+   }
+   ASSERT(blobOffset == endOff);
+   return ML_OK;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoader_Process --
+ *
+ *      Builds and populates a monitor address space per the given a header and
+ *      VCPU count.  Calls out to support functions.
+ *
+ * Result:
+ *      ML_OK if successful, some other value on error.  If an error occurs,
+ *      *line contains the line in the table at which the failure occurred and
+ *      *vcpu contains the VCPU for which processing was running.
+ *
+ * Side effects:
+ *      An address space and page table mappings are created or verified, for
+ *      all VCPUs.  Callouts are called.
+ *
+ *----------------------------------------------------------------------
+ */
+MonLoaderError
+MonLoader_Process(MonLoaderHeader  *header,   // IN/OUT
+                  unsigned          numVCPUs, // IN
+                  void             *args,     // IN
+                  unsigned         *line,     // OUT
+                  Vcpuid           *vcpu)     // OUT
+{
+   unsigned i;
+   MonLoaderError ret;
+   MonLoaderContext ctx;
+
+   if (numVCPUs > MAX_VCPUS) {
+      return ML_ERROR_ARGS;
+   }
+
+   if (header->magic != LOADER_HEADER_MAGIC) {
+      return ML_ERROR_MAGIC;
+   }
+
+   if (vmx86_debug && !line) {
+      return ML_ERROR_ARGS;
+   }
+   *line = LINE_INVALID;
+
+   if (header->entries == 0 || header->count == 0) {
+      return ML_ERROR_TABLE_MISSING;
+   }
+
+   if (!MonLoaderCallout_Init(args, &ctx.envCtx, numVCPUs)) {
+      return ML_ERROR_CALLOUT_INIT;
+   }
+   ret = ML_OK;
+   for (*vcpu = 0; *vcpu < numVCPUs && ret == ML_OK; (*vcpu) += 1) {
+      /* Reset VCPU-specific part of context. */
+      memset(&ctx.vcpu, 0, sizeof ctx.vcpu);
+      ctx.vcpu.currentVCPU = *vcpu;
+      ctx.vcpu.hasAddrSpace = FALSE;
+      for (i = 0; i < header->count && ret == ML_OK; i++) {
+         MonLoaderEntry *entry = &header->entries[i];
+         MonLoaderContentType content    = entry->content;
+         MonLoaderSourceType  source     = entry->source;
+         VPN                  monVPN     = entry->monVPN;
+         uint64               blobOffset = entry->blobSrc.offset;
+         uint64               blobSize   = entry->blobSrc.size;
+         uint64               flags      = entry->flags;
+         uint64               monPages   = entry->monPages;
+         uint64               monBytes   = PAGES_2_BYTES(monPages);
+         uint64               subIndex   = entry->subIndex;
+         uint64               bspOnly    = entry->bspOnly;
+         *line = i;
+
+         /*
+          * The entry is specific to the bootstrap, which only runs in VCPU 0.
+          */
+         if (bspOnly && !IS_BOOT_VCPUID(*vcpu)) {
+            continue;
+         }
+         if (monPages == 0 || monVPN + monPages - 1 > VPN_MAX) {
+            ret = ML_ERROR_SIZE;
+            continue;
+         }
+
+         switch (content) {
+            case ML_CONTENT_ADDRSPACE:
+               if (ctx.vcpu.hasAddrSpace) {
+                  ret = ML_ERROR_DUPLICATE;
+                  continue;
+               }
+               /* Create or verify the address space and page table MPNs. */
+               ret = MonLoaderCreateAddressSpace(&ctx, monVPN, flags, monPages);
+               break;
+            case ML_CONTENT_PAGETABLE_L4:
+            case ML_CONTENT_PAGETABLE_L3:
+            case ML_CONTENT_PAGETABLE_L2:
+            case ML_CONTENT_PAGETABLE_L1:
+               /* Create or verify page table mappings. */
+               ret = MonLoaderMapPageTables(&ctx, CONTENT_TO_PTLEVEL(content),
+                                            flags, monVPN, monPages);
+               break;
+            case ML_CONTENT_ALLOCZERO: {
+               uint64 numPages;
+               if (source == ML_SOURCE_NONE) {
+                  numPages = monPages;
+               } else if (source == ML_SOURCE_BLOB) {
+                  /*
+                   * Used for loading the uninitialized data section
+                   * (.bss), which has a blob size but no blob image
+                   * associated with it.
+                   */
+                  numPages = CEILING(blobSize, PAGE_SIZE);
+               } else {
+                  ret = ML_ERROR_SOURCE_INVALID;
+                  break;
+               }
+               ret = MonLoaderZero(&ctx, flags, monVPN, numPages, monPages,
+                                   &entry->allocs);
+               break;
+            }
+            case ML_CONTENT_COPY:
+               switch (source) {
+                  case ML_SOURCE_BLOB:
+                     ret = MonLoaderCopyFromBlob(&ctx, flags, monVPN, monBytes,
+                                                 blobOffset, blobSize,
+                                                 &entry->allocs);
+                     break;
+                  default:
+                     ret = ML_ERROR_SOURCE_INVALID;
+                     break;
+               }
+               break;
+            case ML_CONTENT_SHARE:
+               switch (source) {
+                  case ML_SOURCE_USER:
+                     ret = MonLoaderShareFromUser(&ctx, flags, monVPN,
+                                                  monPages, subIndex);
+                     break;
+                  case ML_SOURCE_HOST:
+                     ret = MonLoaderShareFromHost(&ctx, flags, monVPN,
+                                                  monPages, subIndex);
+                     break;
+                  case ML_SOURCE_BLOB:
+                     ret = MonLoaderShareFromBlob(&ctx, flags, monVPN,
+                                                  monBytes, blobOffset,
+                                                  blobSize);
+                     break;
+                  default:
+                     ret = ML_ERROR_SOURCE_INVALID;
+                     break;
+               }
+               break;
+            case ML_CONTENT_INVALID:
+            default:
+               ret = ML_ERROR_CONTENT_INVALID;
+               break;
+         }
+      }
+   }
+#ifdef VMKERNEL
+   if (ret == ML_OK) {
+      if (ctx.vcpu.ASLastVPN == 0) {
+         ret = ML_ERROR_NO_ADDRSPACE;
+      }
+   }
+#endif
+   MonLoaderCallout_CleanUp(ctx.envCtx);
+   return ret;
+}

vmmon-only/bootstrap/monLoaderVmmon.c

@@ -0,0 +1,507 @@
+/*********************************************************
+ * Copyright (C) 2016-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * monLoaderVmmon.c --
+ *
+ *      vmmon implementation of the MonLoader callouts.
+ *
+ */
+
+#ifdef __linux__
+#   include "driver-config.h"
+#   include <linux/string.h>
+#else
+#   include <string.h>
+#endif
+
+#include "vm_assert.h"
+#include "vcpuid.h"
+#include "x86types.h"
+#include "hostif.h"
+#include "x86paging_common.h"
+#include "x86paging_64.h"
+#include "vmx86.h"
+#include "monLoader.h"
+#include "monLoaderLog.h"
+#include "vmmblob.h"
+#include "memtrack.h"
+
+typedef struct MonLoaderEnvContext {
+   VMDriver *vm;
+   VMSharedRegion *shRegions;
+} MonLoaderEnvContext;
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCallout_Init --
+ *
+ *      Initializes platform-specific MonLoader context.
+ *
+ * Returns:
+ *      TRUE if successful, FALSE if not.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+Bool
+MonLoaderCallout_Init(void *args, MonLoaderEnvContext **ctx, unsigned numVCPUs)
+{
+   MonLoaderEnvContext *c;
+   MonLoaderArgs *mlArgs = (MonLoaderArgs *)args;
+
+   c = HostIF_AllocKernelMem(sizeof *c, FALSE);
+   if (c == NULL) {
+      Log("Failed to allocate memory for MonLoader context\n");
+      return FALSE;
+   }
+   memset(c, 0, sizeof *c);
+   c->vm = mlArgs->vm;
+   c->shRegions = mlArgs->shRegions;
+
+   *ctx = c;
+   return TRUE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCallout_Cleanup --
+ *
+ *      Cleans up platform-specific MonLoader context.
+ *
+ * Returns:
+ *      None.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+void
+MonLoaderCallout_CleanUp(MonLoaderEnvContext *ctx)
+{
+   ASSERT(ctx != NULL);
+   Vmx86_CleanupVMMPages(ctx->vm);
+   HostIF_FreeKernelMem(ctx);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCallout_GetPageRoot --
+ *
+ *      Gets the page root MPN for a VCPU.
+ *
+ * Returns:
+ *      An MPN, or INVALID_MPN in case of error.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+
+MPN
+MonLoaderCallout_GetPageRoot(MonLoaderEnvContext *ctx,  // IN
+                             Vcpuid               vcpu) // IN
+{
+   VMDriver *vm = ctx->vm;
+
+   if (vcpu >= vm->numVCPUs) {
+      return INVALID_MPN;
+   }
+   return vm->ptRootMpns[vcpu];
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCallout_GetPTE --
+ *
+ *      Gets the page table entry at index idx in a page table MPN.
+ *
+ * Returns:
+ *      TRUE and sets *pte on success.  FALSE otherwise.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+Bool
+MonLoaderCallout_GetPTE(MonLoaderEnvContext *ctx,   // IN
+                        MPN                  ptMPN, // IN
+                        unsigned             idx,   // IN
+                        Vcpuid               vcpu,  // IN
+                        PT_L1E              *pte)   // OUT
+{
+   VMDriver *vm = ctx->vm;
+   MA pteMA = MPN_2_MA(ptMPN) + idx * sizeof *pte;
+
+   ASSERT(idx < (PAGE_SIZE / sizeof *pte));
+   if (HostIF_ReadPhysical(vm, pteMA, PtrToVA64(pte), TRUE, sizeof *pte) != 0) {
+      Log("Failed to read PTE %d from MPN %"FMT64"x\n", idx, ptMPN);
+      return FALSE;
+   }
+   return TRUE;
+}
+
+
+Bool
+MonLoaderCallout_ImportPage(MonLoaderEnvContext *ctx, // IN
+                            MPN mpn,                  // IN
+                            Vcpuid vcpu)              // IN
+{
+   return TRUE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCallout_AllocMPN --
+ *
+ *      Allocates and maps a new VMM page for the specified VCPU.
+ *
+ * Returns:
+ *      An MPN on success, INVALID_MPN on failure.
+ *
+ * Side effects:
+ *      The allocated page is accounted for and its mapping tracked.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MPN
+MonLoaderCallout_AllocMPN(MonLoaderEnvContext *ctx,  // IN
+                          Vcpuid               vcpu) // IN
+{
+   VMDriver *vm = ctx->vm;
+   MemTrackEntry *entry;
+   MPN mpn;
+   VPN vpn;
+
+   if (Vmx86_AllocLockedPages(vm, (VA64)&mpn, 1, TRUE, FALSE) != 1) {
+      Log("Failed to allocate page\n");
+      return INVALID_MPN;
+   }
+   vpn = Vmx86_MapPage(mpn);
+   if (vpn == 0) {
+      Log("Failed to map MPN 0x%"FMT64"x\n", mpn);
+      Vmx86_FreeLockedPages(vm, &mpn, 1);
+      return INVALID_MPN;
+   }
+   HostIF_VMLock(vm, 41);
+   entry = MemTrack_Add(vm->vmmTracker, vpn, mpn);
+   HostIF_VMUnlock(vm, 41);
+   if (entry == NULL) {
+      Log("Failed to track mapping from VPN 0x%"FMTVPN"x to MPN 0x%"FMT64"x\n",
+          vpn, mpn);
+      Vmx86_UnmapPage(vpn);
+      Vmx86_FreeLockedPages(vm, &mpn, 1);
+      return INVALID_MPN;
+   }
+   return mpn;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCallout_MapMPNInPTE --
+ *
+ *      Maps an MPN into a page table at the specified index.
+ *
+ * Returns:
+ *      TRUE on success, FALSE on failure.
+ *
+ * Side effects:
+ *      A mapping is created.
+ *
+ *----------------------------------------------------------------------
+ */
+
+Bool
+MonLoaderCallout_MapMPNInPTE(MonLoaderEnvContext *ctx, // IN
+                             MPN ptMPN,                // IN
+                             unsigned idx,             // IN
+                             uint64 flags,             // IN
+                             MPN mpn,                  // IN
+                             Vcpuid vcpu)              // IN
+{
+   VMDriver *vm = ctx->vm;
+   PT_L1E pte = LM_MAKE_PTE(mpn, 0, flags);
+   MA pteMA = MPN_2_MA(ptMPN) + idx * sizeof pte;
+
+   ASSERT(idx < (PAGE_SIZE / sizeof pte));
+   if (HostIF_WritePhysical(vm, pteMA, PtrToVA64(&pte), TRUE, sizeof pte) !=
+       0) {
+      Warning("Failed to map MPN %"FMT64"x\n", mpn);
+      return FALSE;
+   }
+   LOG(5, "MonLoader mapped MPN %"FMT64"x at PT index %u for VCPU %u\n",
+       mpn, idx, vcpu);
+   return TRUE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCallout_FillPage --
+ *
+ *      Fills a page with a pattern, given the MPN of the page.
+ *
+ * Returns:
+ *      TRUE on success, FALSE if a mapping of the page was not found or
+ *      invalid.
+ *
+ * Side effects:
+ *      The page is filled.
+ *
+ *----------------------------------------------------------------------
+ */
+
+Bool
+MonLoaderCallout_FillPage(MonLoaderEnvContext *ctx,     // IN
+                          uint8                pattern, // IN
+                          MPN                  mpn,     // IN
+                          Vcpuid               vcpu)    // IN
+{
+   VMDriver *vm = ctx->vm;
+   MemTrackEntry *entry;
+
+   HostIF_VMLock(vm, 42);
+   entry = MemTrack_LookupMPN(vm->vmmTracker, mpn);
+   HostIF_VMUnlock(vm, 42);
+   if (entry == NULL || entry->mpn != mpn || entry->vpn == 0) {
+      Log("Failed to look up MPN 0x%"FMT64"x\n", mpn);
+      return FALSE;
+   }
+   memset((void *)VPN_2_VA(entry->vpn), pattern, PAGE_SIZE);
+   return TRUE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCallout_CopyFromBlob --
+ *
+ *     Copies contents of up to PAGE_SIZE length from the blob at a
+ *     given offset into the page specified by the given MPN. Zero-fills
+ *     the remaining space.
+ *
+ * Returns:
+ *      TRUE on success, FALSE on failure.
+ *
+ * Side effects:
+ *      The page is filled with blob contents.
+ *
+ *----------------------------------------------------------------------
+ */
+
+Bool
+MonLoaderCallout_CopyFromBlob(MonLoaderEnvContext *ctx,        // IN
+                              uint64               blobOffset, // IN
+                              size_t               copySize,   // IN
+                              MPN                  mpn,        // IN
+                              Vcpuid               vcpu)       // IN
+{
+   MemTrackEntry *entry;
+   VMDriver *vm = ctx->vm;
+   uint64 blobSize = VmmBlob_GetSize(vm);
+   uint8 *blob = VmmBlob_GetPtr(vm);
+   uint8 *buf;
+
+   if (copySize > PAGE_SIZE || copySize == 0 || blobSize < copySize ||
+       blobOffset > blobSize - copySize) {
+      Log("Invalid VMM blob copy parameters: blobOffset 0x%"FMT64"x, "
+          "copySize 0x%"FMTSZ"x, blobSize 0x%"FMT64"x\n", blobOffset, copySize,
+          blobSize);
+      return FALSE;
+   }
+   HostIF_VMLock(vm, 44);
+   entry = MemTrack_LookupMPN(vm->vmmTracker, mpn);
+   HostIF_VMUnlock(vm, 44);
+   if (entry == NULL || entry->vpn == 0) {
+      Log("Failed to look up MPN 0x%"FMT64"x\n", mpn);
+      return FALSE;
+   }
+   buf = (uint8 *)VPN_2_VA(entry->vpn);
+   memcpy(buf, blob + blobOffset, copySize);
+   memset(buf + copySize, 0, PAGE_SIZE - copySize);
+   return TRUE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderFindSharedRegion --
+ *
+ *      Searches the shared region table for the given index and returns
+ *      the corresponding descriptor.
+ *
+ * Returns:
+ *      A pointer to the shared region descriptor or NULL if not found.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static VMSharedRegion *
+MonLoaderFindSharedRegion(MonLoaderEnvContext *ctx,
+                          uint64 index)
+{
+   int i;
+
+   for (i = 0; i < ML_SHARED_REGIONS_MAX; i++) {
+      if (ctx->shRegions[i].index == index &&
+          ctx->shRegions[i].baseVpn != INVALID_VPN) {
+         return &ctx->shRegions[i];
+      }
+   }
+   return NULL;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderGetSharedRegionMPN --
+ *
+ *      Gets shared page's MPN. Assumes vcpuid == 0 for the bootstrap.
+ *
+ * Returns:
+ *      An MPN or INVALID_MPN.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static MPN
+MonLoaderGetSharedRegionMPN(MonLoaderEnvContext *ctx,
+                            uint64 index,
+                            Vcpuid vcpuid,
+                            unsigned pgOffset)
+{
+   VMSharedRegion *s;
+   VA addr;
+   MPN mpn;
+
+   ASSERT(IS_BOOT_VCPUID(vcpuid));
+
+   s = MonLoaderFindSharedRegion(ctx, index);
+   if (s == NULL) {
+      return INVALID_MPN;
+   }
+   if (pgOffset >= s->numPages) {
+      return INVALID_MPN;
+   }
+   addr = VPN_2_VA(s->baseVpn) + pgOffset * PAGE_SIZE;
+   HostIF_VMLock(ctx->vm, 43);
+   if (HostIF_LookupUserMPN(ctx->vm, addr, &mpn) != PAGE_LOOKUP_SUCCESS) {
+      HostIF_VMUnlock(ctx->vm, 43);
+      Log("Failed to lookup MPN for shared region VA %"FMTVA"x\n", addr);
+      return INVALID_MPN;
+   }
+   HostIF_VMUnlock(ctx->vm, 43);
+   return mpn;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCallout_GetSharedUserPage --
+ *
+ *      Gets a user-provided shared page's MPN for a VCPU.
+ *
+ * Returns:
+ *      An MPN or INVALID_MPN.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MPN
+MonLoaderCallout_GetSharedUserPage(MonLoaderEnvContext *ctx, // IN
+                                   uint64 subIndex,          // IN
+                                   unsigned page,            // IN
+                                   Vcpuid vcpu)              // IN
+{
+   if (subIndex == MONLOADER_HEADER_IDX) {
+      return VmmBlob_GetHeaderMpn(ctx->vm);
+   } else {
+      return MonLoaderGetSharedRegionMPN(ctx, subIndex, vcpu, page);
+   }
+}
+
+
+MPN
+MonLoaderCallout_GetSharedHostPage(MonLoaderEnvContext *ctx,      // IN
+                                   uint64               subIndex, // IN
+                                   unsigned             page,     // IN
+                                   Vcpuid               vcpu)     // IN
+{
+   return MonLoaderGetSharedRegionMPN(ctx, subIndex, vcpu, page);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoaderCallout_GetBlobMpn --
+ *
+ *      Returns the MPN backing the given VMM blob offset.
+ *
+ * Returns:
+ *      An MPN, or INVALID_MPN
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MPN
+MonLoaderCallout_GetBlobMpn(MonLoaderEnvContext *ctx,    // IN
+                            uint64               offset) // IN
+{
+   ASSERT((offset & (PAGE_SIZE - 1)) == 0);
+   return VmmBlob_GetMpn(ctx->vm, BYTES_2_PAGES(offset));
+}

vmmon-only/bootstrap/vmmblob.c

@@ -0,0 +1,231 @@
+/*********************************************************
+ * Copyright (C) 2017-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vmmblob.c --
+ *
+ *     VMM blob management.
+ */
+
+#ifdef __linux__
+#   include "driver-config.h"
+#endif
+#include "vmware.h"
+#include "vm_assert.h"
+#include "hostif.h"
+#include "vmmblob.h"
+#include "monLoader.h"
+
+#define VMMBLOB_SIZE_MAX (22 * 1024 * 1024) /* Ensure enough space for
+                                             * obj build with * GCOV_VMM=1. */
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VmmBlob_GetPtr --
+ *
+ *      Returns a pointer to the buffer containing the VMM blob.
+ *
+ * Results:
+ *      A pointer.
+ *
+ *----------------------------------------------------------------------
+ */
+
+uint8 *
+VmmBlob_GetPtr(VMDriver *vm)
+{
+   return vm->blobInfo->blobPtr;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VmmBlob_GetSize --
+ *
+ *      Returns the size of the VMM blob in bytes.
+ *
+ * Results:
+ *      Number of bytes.
+ *
+ *----------------------------------------------------------------------
+ */
+
+uint64
+VmmBlob_GetSize(VMDriver *vm)
+{
+   return vm->blobInfo->numBytes;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VmmBlob_GetMpn --
+ *
+ *      Returns the MPN backing a VMM blob page.
+ *
+ * Results:
+ *      An MPN, or INVALID_MPN.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MPN
+VmmBlob_GetMpn(VMDriver *vm, uint64 pgOffset)
+{
+   VmmBlobInfo *bi = vm->blobInfo;
+   uint32 blobNumPages = CEILING(bi->numBytes, PAGE_SIZE);
+   MPN mpn = INVALID_MPN;
+
+   if (pgOffset < blobNumPages) {
+      mpn = bi->mpns[pgOffset];
+   }
+   return mpn;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VmmBlob_GetHeaderMpn --
+ *
+ *      Returns the MPN backing the MonLoader header.
+ *
+ * Results:
+ *      An MPN, or INVALID_MPN.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MPN
+VmmBlob_GetHeaderMpn(VMDriver *vm)
+{
+   VmmBlobInfo *bi = vm->blobInfo;
+   uint64 headerOffset = (uint8 *)bi->header - (uint8 *)bi->blobPtr;
+   ASSERT((headerOffset & (PAGE_SIZE - 1)) == 0);
+   return VmmBlob_GetMpn(vm, BYTES_2_PAGES(headerOffset));
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VmmBlob_Cleanup --
+ *
+ *      Cleans up VMM blob state by freeing its memory and associated
+ *      metadata.
+ *
+ * Results:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+void
+VmmBlob_Cleanup(VmmBlobInfo *bi)
+{
+   const uint32 numPages = CEILING(bi->numBytes, PAGE_SIZE);
+
+   HostIF_FreeKernelPages(numPages, bi->blobPtr);
+   HostIF_FreeKernelMem(bi->mpns);
+   HostIF_FreeKernelMem(bi);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VmmBlob_Load --
+ *
+ *      Copies and instantiates a VMM bootstrap blob from userspace.
+ *
+ * Results:
+ *      TRUE if successful, FALSE if not.
+ *
+ *----------------------------------------------------------------------
+ */
+
+Bool
+VmmBlob_Load(UserVA64 blobAddr,
+             uint32 numBytes,
+             uint32 headerOffset,
+             VmmBlobInfo **blobInfo)
+{
+   uint8 *blob = NULL;
+   MPN *mpns = NULL;
+   MonLoaderHeader *header;
+   VmmBlobInfo *bi = NULL;
+   uint32 numPages = CEILING(numBytes, PAGE_SIZE);
+   size_t fixedHdrSize = MonLoader_GetFixedHeaderSize();
+
+   if (numBytes > VMMBLOB_SIZE_MAX || headerOffset > numBytes ||
+       fixedHdrSize > numBytes - headerOffset) {
+      Warning("Invalid arguments for processing bootstrap. "
+              "Header offset: %u, Fixed header size: %"FMTSZ"u bytes, "
+              "Blob size: %u bytes\n", headerOffset, fixedHdrSize, numBytes);
+      goto error;
+   }
+   mpns = HostIF_AllocKernelMem(numPages * sizeof(*mpns), FALSE);
+   if (mpns == NULL) {
+      Warning("Memory allocation for VMM bootstrap blob failed\n");
+      goto error;
+   }
+   bi = HostIF_AllocKernelMem(sizeof *bi, FALSE);
+   if (bi == NULL) {
+      Warning("Memory allocation for VMM bootstrap blob failed\n");
+      goto error;
+   }
+   blob = HostIF_AllocKernelPages(numPages, mpns);
+   if (blob == NULL) {
+      Warning("Memory allocation for VMM bootstrap blob failed\n");
+      goto error;
+   }
+   if (HostIF_CopyFromUser(blob, blobAddr, numBytes) != 0) {
+      Warning("Error copying VMM bootstrap blob from userspace\n");
+      goto error;
+   }
+   header = (MonLoaderHeader *)(blob + headerOffset);
+   if (header->count > (numBytes - headerOffset - fixedHdrSize) /
+                        sizeof (MonLoaderEntry)) {
+      Warning("Invalid arguments for processing bootstrap. "
+              "Header offset: %u, Number of header entries: %u, "
+              "Blob size: %u bytes\n", headerOffset, header->count, numBytes);
+      goto error;
+   }
+   bi->mpns = mpns;
+   bi->blobPtr = blob;
+   bi->numBytes = numBytes;
+   bi->header = header;
+   *blobInfo = bi;
+
+   return TRUE;
+
+error:
+   if (blob != NULL) {
+      HostIF_FreeKernelPages(numPages, blob);
+   }
+   if (bi != NULL) {
+      HostIF_FreeKernelMem(bi);
+   }
+   if (mpns != NULL) {
+      HostIF_FreeKernelMem(mpns);
+   }
+   return FALSE;
+}

vmmon-only/common/apic.c

@@ -0,0 +1,278 @@
+/*********************************************************
+ * Copyright (C) 2011, 2016, 2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#include "vmware.h"
+#include "x86apic.h"
+#include "x86msr.h"
+#include "x86cpuid_asm.h"
+#include "vm_asm.h"
+#include "cpuid.h"
+#include "apic.h"
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * APIC_GetMA --
+ *
+ *      Return the MA of the host's APIC by reading the APIC_BASE
+ *      MSR and applying any necessary masking.
+ *
+ * Side effects:
+ *      None
+ *
+ * Return value:
+ *      MA of host APIC if successful (guaranteed to be page-aligned),
+ *      or the sentinel (MA)-1 if unsucessful or if X2 APIC mode is enabled
+ *      since this disables the MMIO interface.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MA
+APIC_GetMA(void)
+{
+   uint64 result;
+   CpuidVendor cpuVendor = CPUID_GetVendor();
+   uint32 features = CPUID_GetFeatures();
+
+   if (!CPUID_ISSET(1, EDX, MSR, features) ||
+       !CPUID_ISSET(1, EDX, APIC, features)) {
+      return (MA)-1;
+   }
+
+   if (cpuVendor != CPUID_VENDOR_INTEL &&
+       cpuVendor != CPUID_VENDOR_AMD &&
+       cpuVendor != CPUID_VENDOR_HYGON) {
+      return (MA)-1;
+   }
+
+   /*
+    * Check if X2 APIC mode is enabled.
+    */
+
+   if ((X86MSR_GetMSR(MSR_APIC_BASE) & APIC_MSR_X2APIC_ENABLED) != 0) {
+      return (MA)-1;
+   }
+
+   /*
+    * APIC is present and enabled.  The CPUID[0x1].edx[APIC] bit,
+    * already checked, mirrors the APIC base MSR's enable bit.
+    */
+
+   // Mask out goo in the low 12 bits, which is unrelated to the address.
+   result = X86MSR_GetMSR(MSR_APIC_BASE) & ~MASK64(PAGE_SHIFT);
+
+   /*
+    * On Intel, the high bits are reserved so we mask.
+    * On AMD and Hygon, high bits are explicitly MBZ, so no need.
+    */
+   if (cpuVendor == CPUID_VENDOR_INTEL) {
+      /*
+       * Intel suggests using CPUID 0x80000008.eax[7-0] (physical
+       * address size), with 36 (24 bit MPNs) as a fallback.
+       */
+      unsigned numPhysicalBits = 36;
+
+      if (CPUID_AddressSizeSupported()) {
+         numPhysicalBits = __GET_EAX_FROM_CPUID(0x80000008) & 0xff;
+      }
+
+      result &= MASK64(numPhysicalBits);
+   }
+
+   ASSERT_ON_COMPILE(sizeof(result) == sizeof(MA));
+   return result;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * APIC_Read --
+ *
+ *      Reads the given APIC register using the proper interface.  Does not
+ *      check to see if the register number is valid.
+ *
+ * Side effects:
+ *      Yes.
+ *
+ * Return value:
+ *      Value of the register.
+ *
+ *----------------------------------------------------------------------
+ */
+
+uint32
+APIC_Read(const APICDescriptor *desc, // IN
+          int regNum)                 // IN
+{
+   if (desc->isX2) {
+      return (uint32)X86MSR_GetMSR(MSR_X2APIC_BASE + regNum);
+   } else {
+      return desc->base[regNum][0];
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * APIC_ReadID --
+ *
+ *      Reads the APIC ID using the proper interface.  The semantics of the
+ *      ID are different in X2APIC mode so APIC_Read() should not be used.
+ *
+ * Side effects:
+ *      None.
+ *
+ * Return value:
+ *      APIC ID.
+ *
+ *----------------------------------------------------------------------
+ */
+
+uint32
+APIC_ReadID(const APICDescriptor *desc) // IN
+{
+   uint32 reg = APIC_Read(desc, APICR_ID);
+
+   if (desc->isX2) {
+      return reg;
+   } else {
+      return (reg & XAPIC_ID_MASK) >> APIC_ID_SHIFT;
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * APIC_Write --
+ *
+ *      Writes the given value to the given APIC register using the proper
+ *      interface.  Does not check to see if the register number is valid.
+ *
+ * Side effects:
+ *      Yes.
+ *
+ * Return value:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+void
+APIC_Write(const APICDescriptor *desc, // IN
+           int regNum,                 // IN
+           uint32 val)                 // IN
+{
+   if (desc->isX2) {
+      X86MSR_SetMSR(MSR_X2APIC_BASE + regNum, val);
+   } else {
+      desc->base[regNum][0] = val;
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * APIC_ReadICR --
+ *
+ *      Reads the APIC ICR using the proper interface.  The semantics of the
+ *      ICR are different in X2APIC mode so APIC_Read() should not be used.
+ *
+ * Side effects:
+ *      Yes.
+ *
+ * Return value:
+ *      The full 64-bit value of the ICR.
+ *
+ *----------------------------------------------------------------------
+ */
+
+uint64
+APIC_ReadICR(const APICDescriptor *desc) // IN
+{
+   if (desc->isX2) {
+      return X86MSR_GetMSR(MSR_X2APIC_BASE + APICR_ICRLO);
+   } else {
+      uint32 icrHi = desc->base[APICR_ICRHI][0];
+      uint32 icrLo = desc->base[APICR_ICRLO][0];
+      return (uint64) icrHi << 32 | icrLo;
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * APIC_WriteICR --
+ *
+ *      Writes the given value to the APIC ICR using the proper interface.
+ *      The semantics of the ICR are different in X2APIC mode so APIC_Write()
+ *      should not be used.
+ *
+ * Side effects:
+ *      Yes.
+ *
+ * Return value:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+void
+APIC_WriteICR(const APICDescriptor *desc, // IN
+              uint32 id,                  // IN
+              uint32 icrLo)               // IN
+{
+   if (desc->isX2) {
+      uint64 icr = (uint64) id << 32 | icrLo;
+      X86MSR_SetMSR(MSR_X2APIC_BASE + APICR_ICRLO, icr);
+   } else {
+      ASSERT(!(id & ~(APIC_ICRHI_DEST_MASK >> APIC_ICRHI_DEST_OFFSET)));
+      desc->base[APICR_ICRHI][0] = id << APIC_ICRHI_DEST_OFFSET;
+      desc->base[APICR_ICRLO][0] = icrLo;
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * APIC_MaxLvt --
+ *
+ *      Reads the maximum number of LVT entries from the APIC version register.
+ *
+ * Side effects:
+ *      No.
+ *
+ * Return value:
+ *      The maximum number of LVT entries.
+ *
+ *----------------------------------------------------------------------
+ */
+
+uint32
+APIC_MaxLVT(const APICDescriptor *desc) // IN
+{
+   uint32 ver = APIC_Read(desc, APICR_VERSION);
+
+   return (ver >> APIC_MAX_LVT_SHIFT) & APIC_MAX_LVT_MASK;
+}

vmmon-only/common/apic.h

@@ -0,0 +1,47 @@
+/*********************************************************
+ * Copyright (C) 2011 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * apic.h --
+ *
+ *      Some host APIC helper functions
+ */
+
+#ifndef APIC_H
+#define APIC_H
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#include "vm_basic_types.h"
+
+typedef struct {
+   Bool isX2;
+   volatile uint32 (*base)[4];
+} APICDescriptor;
+
+MA APIC_GetMA(void);
+uint32 APIC_Read(const APICDescriptor *desc, int regNum);
+void APIC_Write(const APICDescriptor *desc, int regNum, uint32 val);
+void APIC_WriteICR(const APICDescriptor *desc, uint32 id, uint32 icrLo);
+uint64 APIC_ReadICR(const APICDescriptor *desc);
+uint32 APIC_ReadID(const APICDescriptor *desc);
+uint32 APIC_MaxLVT(const APICDescriptor *desc);
+
+#endif

vmmon-only/common/comport.c

@@ -0,0 +1,203 @@
+/*********************************************************
+ * Copyright (C) 2006,2016 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * comport.c --
+ *
+ *      Simple COM1 port routines for debugging code that can't do any real
+ *      host IO, such as the worldswitch and related.
+ *
+ *      They all wait for the last character to complete transmission so if the
+ *      system crashes immediately on return, the last character will be seen
+ *      by the remote end.
+ *
+ *      These routines do not have any external dependencies so can be called
+ *      by any code that has privilege to access IO ports.
+ *
+ *      Under Windows, they can be made to forward output to DbgPrint for
+ *      printing via the debugger.  Just have USE_DBGPRINT set to 1.  If you
+ *      let USE_DBGPRINT be 0 with Windows, make sure the comport hardware is
+ *      powered-up by leaving Hyperterm running with the comport open, else
+ *      Windows will power the chip down.
+ */
+
+#include "comport.h"
+#include "vm_basic_asm.h"  // for INB, OUTB
+
+#if 000 // defined(_MSC_VER)
+#define USE_DBGPRINT 1
+#define USE_MACPORT80 0
+#else
+#define USE_DBGPRINT 0
+#ifdef __APPLE__
+#define USE_MACPORT80 1
+#else
+#define USE_MACPORT80 0
+#endif
+#endif
+
+#if USE_DBGPRINT
+void DbgPrint(char const *format, ...);
+#elif !USE_MACPORT80
+#define IOBASE 0x3F8  // COM1 base IO port number
+#define BAUD 115200   // baud rate
+#define THR 0         // transmitter holding register
+#define LSR 5         // line status register
+#define LSR_TE 0x20   // - transmit fifo completely empty
+#define LSR_TI 0x40   // - transmitter idle
+#endif
+
+
+void
+CP_Init(void)
+{
+#if !USE_DBGPRINT && !USE_MACPORT80
+   OUTB(IOBASE+3, 0x83);               // LCR=select DLL/DLH, wordlen=8 bits
+   OUTB(IOBASE+0, (115200/BAUD)&255);  // DLL=lo order baud rate
+   OUTB(IOBASE+1, (115200/BAUD)/256);  // DLH=hi order baud rate
+   OUTB(IOBASE+3, 0x03);               // LCR=select RBR/THR/IER
+   OUTB(IOBASE+4, 0x07);               // MCR=dtr, rts, port-enable
+   OUTB(IOBASE+2, 0x07);               // FCR=reset rcv fifo, reset xmt fifo
+   OUTB(IOBASE+1, 0);                  // IER=disable all interrupts
+#endif
+}
+
+
+void
+CP_PutChr(uint8 ch) // IN
+{
+#if USE_DBGPRINT
+   DbgPrint("%c", ch);
+#elif USE_MACPORT80
+   int bit;
+
+   OUTB(0x80, (ch & 1) | 0x10);
+   for (bit = 1; bit < 64; bit ++) {
+      OUTB(0x80, (ch >> (bit & 7)) & 1);
+   }
+#else
+   if (ch == '\n') CP_PutChr('\r');
+   while ((INB(IOBASE+LSR) & LSR_TE) == 0) { }
+   OUTB(IOBASE+THR, ch);
+   while ((INB(IOBASE+LSR) & LSR_TI) == 0) { }
+#endif
+}
+
+
+void
+CP_PutDec(uint32 value) // IN
+{
+#if USE_DBGPRINT
+   DbgPrint("%u", value);
+#else
+   char s[12];
+   int i;
+
+   i = 0;
+   do {
+      s[i++] = (value % 10) + '0';
+      value /= 10;
+   } while (value > 0);
+   while (--i >= 0) CP_PutChr(s[i]);
+#endif
+}
+
+
+void
+CP_PutHexPtr(void *value) // IN
+{
+   if (sizeof value == 8) {
+      CP_PutHex64((uint64)(VA)value);
+   }
+   if (sizeof value == 4) {
+      CP_PutHex32((uint32)(VA)value);
+   }
+}
+
+
+void
+CP_PutHex64(uint64 value) // IN
+{
+   CP_PutHex32((uint32)(value >> 32));
+   CP_PutHex32((uint32)value);
+}
+
+
+void
+CP_PutHex32(uint32 value) // IN
+{
+#if USE_DBGPRINT
+   DbgPrint("%8.8X", value);
+#else
+   CP_PutHex16((uint16)(value >> 16));
+   CP_PutHex16((uint16)value);
+#endif
+}
+
+
+void
+CP_PutHex16(uint16 value) // IN
+{
+#if USE_DBGPRINT
+   DbgPrint("%4.4X", value);
+#else
+   CP_PutHex8((uint8)(value >> 8));
+   CP_PutHex8((uint8)value);
+#endif
+}
+
+
+void
+CP_PutHex8(uint8 value) // IN
+{
+#if USE_DBGPRINT
+   DbgPrint("%2.2X", value);
+#else
+   CP_PutChr("0123456789ABCDEF"[(value>>4)&15]);
+   CP_PutChr("0123456789ABCDEF"[value&15]);
+#endif
+}
+
+
+void
+CP_PutSp(void)
+{
+   CP_PutChr(' ');
+}
+
+
+void
+CP_PutCrLf(void)
+{
+   CP_PutChr('\n');
+}
+
+
+void
+CP_PutStr(char const *s) // IN
+{
+#if USE_DBGPRINT
+   DbgPrint("%s", s);
+#else
+   char c;
+
+   while ((c = *(s ++)) != 0) {
+      CP_PutChr(c);
+   }
+#endif
+}

vmmon-only/common/comport.h

@@ -0,0 +1,41 @@
+/*********************************************************
+ * Copyright (C) 2006 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef _COMPORT_H
+#define _COMPORT_H
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMX
+#include "includeCheck.h"
+
+#include "vm_basic_types.h"  // for uint8, et al
+
+void CP_Init(void);
+void CP_PutChr(uint8 ch);
+void CP_PutDec(uint32 value);
+void CP_PutHexPtr(void *value);
+void CP_PutHex64(uint64 value);
+void CP_PutHex32(uint32 value);
+void CP_PutHex16(uint16 value);
+void CP_PutHex8(uint8 value);
+void CP_PutSp(void);
+void CP_PutCrLf(void);
+void CP_PutStr(char const *s);
+
+#endif

vmmon-only/common/cpuid.c

@@ -0,0 +1,153 @@
+/*********************************************************
+ * Copyright (C) 1998, 2016-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifdef __linux__
+/* Must come before any kernel header file --hpreg */
+#   include "driver-config.h"
+
+#   include <linux/string.h>
+#endif
+#ifdef __APPLE__
+#   include <string.h> // For strcmp().
+#endif
+
+#include "vmware.h"
+#include "vm_assert.h"
+#include "hostif.h"
+#include "cpuid.h"
+#include "x86cpuid_asm.h"
+#include "x86svm.h"
+#include "x86vt.h"
+
+uint32      cpuidFeatures;
+CpuidVendor cpuidVendor = CPUID_NUM_VENDORS;
+uint32      cpuidVersion;
+Bool        hostSupportsVT;
+Bool        hostSupportsSVM;
+Bool        hostHasSpecCtrl;
+Bool        hostSupportsXSave;
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * CPUIDExtendedSupported --
+ *
+ *     Determine whether processor supports extended CPUID (0x8000xxxx)
+ *     and how many of them.
+ *
+ * Results:
+ *     0         if extended CPUID is not supported
+ *     otherwise maximum extended CPUID supported (bit 31 set)
+ *
+ * Side effects:
+ *     None.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static uint32
+CPUIDExtendedSupported(void)
+{
+   uint32 eax;
+
+   eax = __GET_EAX_FROM_CPUID(0x80000000);
+   if ((eax & 0x80000000) != 0x80000000) {
+      return 0;
+   }
+
+   return eax;
+}
+
+
+void
+CPUID_Init(void)
+{
+   CPUIDRegs regs;
+   uint32 *ptr;
+   char name[16];
+
+   __GET_CPUID(1, &regs);
+   cpuidVersion = regs.eax;
+   cpuidFeatures = regs.edx;
+   hostSupportsXSave = CPUID_ISSET(1, ECX, XSAVE, regs.ecx);
+
+   __GET_CPUID(0, &regs);
+   ptr = (uint32 *)name;
+   ptr[0] = regs.ebx;
+   ptr[1] = regs.edx;
+   ptr[2] = regs.ecx;
+   ptr[3] = 0;
+
+   if (strcmp(name, CPUID_INTEL_VENDOR_STRING_FIXED) == 0) {
+      cpuidVendor = CPUID_VENDOR_INTEL;
+   } else if (strcmp(name, CPUID_AMD_VENDOR_STRING_FIXED) == 0) {
+      cpuidVendor = CPUID_VENDOR_AMD;
+   } else if (strcmp(name, CPUID_CYRIX_VENDOR_STRING_FIXED) == 0) {
+      cpuidVendor = CPUID_VENDOR_CYRIX;
+   } else if (strcmp(name, CPUID_HYGON_VENDOR_STRING_FIXED) == 0) {
+      cpuidVendor = CPUID_VENDOR_HYGON;
+   } else {
+      Warning("VMMON CPUID: Unrecognized CPU\n");
+      cpuidVendor = CPUID_VENDOR_UNKNOWN;
+   }
+
+   __GET_CPUID2(7, 0, &regs);
+   hostHasSpecCtrl = cpuidVendor == CPUID_VENDOR_INTEL &&
+                     (CPUID_ISSET(7, EDX, IBRSIBPB, regs.edx) ||
+                      CPUID_ISSET(7, EDX, STIBP, regs.edx));
+
+   hostSupportsVT = VT_CapableCPU();
+   hostSupportsSVM = SVM_CapableCPU();
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * CPUID_AddressSizeSupported --
+ *
+ *     Determine whether processor supports the address size cpuid
+ *     extended leaf.
+ *
+ * Results:
+ *     True iff the processor supports CPUID 0x80000008.
+ *
+ * Side effects:
+ *     It determines value only on first call, caching it for future.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+Bool
+CPUID_AddressSizeSupported(void)
+{
+   /*
+    * It is OK to use local static variables here as 'result' does not depend
+    * on any work done in CPUID_Init(). It purely depends on the CPU.
+    */
+   static Bool initialized = FALSE;
+   static Bool result;
+
+   if (UNLIKELY(!initialized)) {
+      result = CPUIDExtendedSupported() >= 0x80000008;
+      initialized = TRUE;
+   }
+
+   return result;
+}

vmmon-only/common/cpuid.h

@@ -0,0 +1,100 @@
+/*********************************************************
+ * Copyright (C) 1998, 2016-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * cpuid.h --
+ *
+ *    wrap CPUID instruction
+ */
+
+#ifndef CPUID_H
+#define CPUID_H
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#include "x86cpuid.h"
+
+extern uint32      cpuidFeatures;
+extern uint32      cpuidVersion;
+extern CpuidVendor cpuidVendor;
+extern Bool        hostSupportsVT;
+extern Bool        hostSupportsSVM;
+extern Bool        hostHasSpecCtrl;
+extern Bool        hostSupportsXSave;
+
+void CPUID_Init(void);
+Bool CPUID_AddressSizeSupported(void);
+
+static INLINE uint32
+CPUID_GetFeatures(void)
+{
+   return cpuidFeatures;
+}
+
+static INLINE CpuidVendor
+CPUID_GetVendor(void)
+{
+   ASSERT(cpuidVendor != CPUID_NUM_VENDORS);
+   return cpuidVendor;
+}
+
+static INLINE uint32
+CPUID_GetVersion(void)
+{
+   return cpuidVersion;
+}
+
+static INLINE Bool
+CPUID_HostSupportsVT(void)
+{
+   return hostSupportsVT;
+}
+
+static INLINE Bool
+CPUID_HostSupportsSVM(void)
+{
+   return hostSupportsSVM;
+}
+
+static INLINE Bool
+CPUID_HostSupportsHV(void)
+{
+   return hostSupportsVT || hostSupportsSVM;
+}
+
+static INLINE Bool
+CPUID_HostSupportsSpecCtrl(void)
+{
+   return hostHasSpecCtrl;
+}
+
+static INLINE Bool
+CPUID_HostSupportsXSave(void)
+{
+   return hostSupportsXSave;
+}
+
+static INLINE Bool
+CPUID_SSE2Supported(void)
+{
+   return CPUID_ISSET(1, EDX, SSE2, CPUID_GetFeatures());
+}
+
+#endif

vmmon-only/common/hostKernel.h

@@ -0,0 +1,83 @@
+/*********************************************************
+ * Copyright (C) 1998,2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * hostKernel.h --
+ *
+ *    Definition of HOST_KERNEL_* --hpreg
+ */
+
+
+#ifndef __HOST_KERNEL_H__
+#   define __HOST_KERNEL_H__
+
+
+#   ifdef __linux__
+/*
+ * In some cases, this files needs to include Linux kernel header file
+ * asm/page.h.
+ *
+ * However, asm/page.h defines PAGE_SHIFT, PAGE_SIZE, PAGE_MASK, PAGE_OFFSET
+ * and VMware header file vm_basic_types.h defines PAGE_SHIFT, PAGE_SIZE,
+ * PAGE_MASK, PAGE_OFFSET. PAGE_MASK and PAGE_OFFSET are defined differently
+ * (XXX we should really prefix the VMware version with VM_ to prevent any
+ * further confusion), but fortunately the driver does not use them.
+ *
+ * So in this file, we must solve the definition conflict for files that
+ * include both this file and vm_basic_types.h. 2 cases can occur:
+ *
+ * 1) this file is included before vm_basic_types.h is included. This is fine,
+ *    because vm_basic_types.h only defines PAGE_* if they don't exist yet.
+ *
+ * 2) vm_basic_types.h is included before this file is included.
+ * We must undefine
+ *    PAGE_* in between. But this only works if asm/page.h is not included
+ *    before this file is included.
+ *
+ * In summary: if you make sure you do not include asm/page.h before you
+ * include this file, then we guarantee that:
+ * . This file and vm_basic_types.h can be included in any order
+ * . asm/page.h will be included
+ * . The PAGE_* definitions will come from asm/page.h
+ *
+ *   --hpreg
+ */
+
+/* Must come before any kernel header file --hpreg */
+#      include "driver-config.h"
+
+#      undef PAGE_SHIFT
+#      undef PAGE_SIZE
+#      undef PAGE_MASK
+#      undef PAGE_OFFSET
+
+/* For __PAGE_OFFSET --hpreg */
+#      include <asm/page.h>
+
+#      define HOST_KERNEL_VA_2_LA(_x) (_x)
+#      define HOST_KERNEL_LA_2_VA(_x) (_x)
+#  else
+/* For VA and LA --hpreg */
+#      include "vm_basic_types.h"
+
+#      define HOST_KERNEL_VA_2_LA(_addr) ((LA)(_addr))
+#      define HOST_KERNEL_LA_2_VA(_addr) ((VA)(_addr))
+#  endif
+
+
+#endif /* __HOST_KERNEL_H__ */

vmmon-only/common/hostif.h

@@ -0,0 +1,158 @@
+/*********************************************************
+ * Copyright (C) 1998-2020 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ *  hostif.h - Platform dependent interface for supporting
+ *             the vmx86 device driver.
+ */
+
+
+#ifndef _HOSTIF_H_
+#define _HOSTIF_H_
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#include "vmx86.h"
+#include "vcpuset.h"
+
+#include "hostifMem.h"
+#include "hostifGlobalLock.h"
+
+/*
+ * Host-specific definitions.
+ */
+#if !__linux__ && !defined(WINNT_DDK) && !defined __APPLE__
+#error "Only Linux or NT or Mac OS defined for now."
+#endif
+
+
+/*
+ * The default monitor spin time for crosscalls is 50 usec
+ * in vmcore/vmx/main/monitor_init.c.  This value is used
+ * in vmx86_YieldToSet to decide whether to block and wait
+ * for another vCPU to process our crosscall, or just wake
+ * up the other vCPUs and go back to monitor.
+ */
+
+#define CROSSCALL_SPIN_SHORT_US 50
+
+/*
+ * Sleep timeout in usec, see above comment for CROSSCALL_SPIN_SHORT_US
+ */
+
+#define CROSSCALL_SLEEP_US 1000
+
+typedef struct HostIFContigMemMap {
+   MPN mpn;
+   void *addr;
+   PageCnt pages;
+   struct HostIFContigMemMap *next;
+} HostIFContigMemMap;
+
+EXTERN Bool  HostIF_Init(VMDriver *vm, uint32 numVCPUs);
+EXTERN int   HostIF_LookupUserMPN(VMDriver *vm, VA64 uAddr, MPN *mpn);
+EXTERN void *HostIF_MapCrossPage(VMDriver *vm, VA64 uAddr);
+
+EXTERN void *HostIF_AllocPage(void);
+EXTERN void  HostIF_FreePage(void *ptr);
+
+EXTERN VPN   HostIF_MapPage(MPN mpn);
+EXTERN void  HostIF_UnmapPage(VPN vpn);
+EXTERN int   HostIF_LockPage(VMDriver *vm, VA64 uAddr,
+                             Bool allowMultipleMPNsPerVA, MPN *mpn);
+EXTERN int   HostIF_UnlockPage(VMDriver *vm, VA64 uAddr);
+EXTERN int   HostIF_UnlockPageByMPN(VMDriver *vm, MPN mpn, VA64 uAddr);
+EXTERN Bool  HostIF_IsLockedByMPN(VMDriver *vm, MPN mpn);
+EXTERN void  HostIF_FreeAllResources(VMDriver *vm);
+EXTERN uint64 HostIF_ReadUptime(void);
+EXTERN uint64 HostIF_UptimeFrequency(void);
+EXTERN PageCnt HostIF_EstimateLockedPageLimit(const VMDriver *vm,
+                                              PageCnt lockedPages);
+EXTERN void  HostIF_Wait(unsigned int timeoutMs);
+EXTERN void  HostIF_WaitForFreePages(unsigned int timeoutMs);
+EXTERN void *HostIF_AllocKernelPages(PageCnt numPages, MPN *mpns);
+EXTERN void  HostIF_FreeKernelPages(PageCnt numPages, void *ptr);
+EXTERN HostIFContigMemMap *HostIF_AllocContigPages(VMDriver *vm,
+                                                   PageCnt numPages);
+EXTERN void  HostIF_FreeContigPages(VMDriver *vm, HostIFContigMemMap *mapping);
+EXTERN void  HostIF_VMLock(VMDriver *vm, int callerID);
+EXTERN void  HostIF_VMUnlock(VMDriver *vm, int callerID);
+#ifdef VMX86_DEBUG
+EXTERN Bool HostIF_VMLockIsHeld(VMDriver *vm);
+#endif
+
+EXTERN void  HostIF_APICInit(VMDriver *vm);
+EXTERN uint8 HostIF_GetMonitorIPIVector(void);
+EXTERN uint8 HostIF_GetHVIPIVector(void);
+EXTERN void  HostIF_GetTimerVectors(uint8 *v0, uint8 *v1);
+
+EXTERN int   HostIF_SemaphoreWait(VMDriver *vm,
+                                  Vcpuid vcpuid,
+                                  uint64 *args);
+
+EXTERN int   HostIF_SemaphoreSignal(uint64 *args);
+
+EXTERN void  HostIF_SemaphoreForceWakeup(VMDriver *vm, const VCPUSet *vcs);
+EXTERN void  HostIF_IPI(VMDriver *vm, const VCPUSet *vcs);
+EXTERN void  HostIF_OneIPI(VMDriver *vm, Vcpuid v);
+
+EXTERN uint32 HostIF_GetCurrentPCPU(void);
+EXTERN void HostIF_CallOnEachCPU(void (*func)(void *), void *data);
+
+EXTERN Bool HostIF_PrepareWaitForThreads(VMDriver *vm, Vcpuid currVcpu);
+EXTERN void HostIF_WaitForThreads(VMDriver *vm, Vcpuid currVcpu);
+EXTERN void HostIF_CancelWaitForThreads(VMDriver *vm, Vcpuid currVcpu);
+EXTERN void HostIF_WakeUpYielders(VMDriver *vm, Vcpuid currVcpu);
+
+EXTERN int64 HostIF_AllocLockedPages(VMDriver *vm, VA64 addr,
+                                     PageCnt numPages, Bool kernelMPNBuffer);
+EXTERN int HostIF_FreeLockedPages(VMDriver *vm, MPN *mpns, PageCnt numPages);
+EXTERN MPN HostIF_GetNextAnonPage(VMDriver *vm, MPN mpn);
+EXTERN PageCnt HostIF_GetNumAnonPages(VMDriver *vm);
+EXTERN MPN HostIF_AllocLowPage(VMDriver *vm);
+
+EXTERN int HostIF_ReadPhysical(VMDriver *vm, MA ma, VA64 addr,
+                               Bool kernelBuffer, size_t len);
+EXTERN int HostIF_WritePhysical(VMDriver *vm, MA ma, VA64 addr,
+                                Bool kernelBuffer, size_t len);
+EXTERN int HostIF_WriteMachinePage(MPN mpn, VA64 addr);
+#if defined __APPLE__
+// There is no need for a fast clock lock on Mac OS.
+#define HostIF_FastClockLock(_callerID) do {} while (0)
+#define HostIF_FastClockUnlock(_callerID) do {} while (0)
+#else
+EXTERN void HostIF_FastClockLock(int callerID);
+EXTERN void HostIF_FastClockUnlock(int callerID);
+#endif
+EXTERN int HostIF_SetFastClockRate(unsigned rate);
+
+EXTERN MPN HostIF_AllocMachinePage(void);
+EXTERN void HostIF_FreeMachinePage(MPN mpn);
+
+EXTERN int HostIF_SafeRDMSR(uint32 msr, uint64 *val);
+
+EXTERN int HostIF_CopyFromUser(void *dst, VA64 src, size_t len);
+EXTERN int HostIF_CopyToUser(VA64 dst, const void *src, size_t len);
+
+#if defined __APPLE__
+EXTERN void HostIF_PageUnitTest(void);
+#endif
+
+#endif // ifdef _HOSTIF_H_

vmmon-only/common/hostifGlobalLock.h

@@ -0,0 +1,45 @@
+/*********************************************************
+ * Copyright (C) 1998 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ *  hostifGlobalLock.h - Platform dependent interface. This module
+ *                       defines functions for manipulating/checking
+ *                       the Global lock used by some drivers.
+ */
+
+
+#ifndef _HOSTIFGLOBALLOCK_H_
+#define _HOSTIFGLOBALLOCK_H_
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+
+#ifdef __APPLE__
+Bool HostIFGlobalLock_Start(void);
+void HostIFGlobalLock_Stop(void);
+#endif
+void HostIF_GlobalLock(int callerID);
+void HostIF_GlobalUnlock(int callerID);
+#ifdef VMX86_DEBUG
+Bool HostIF_GlobalLockIsHeld(void);
+#endif
+
+
+#endif // ifdef _HOSTIFGLOBALLOCK_H_

vmmon-only/common/hostifMem.h

@@ -0,0 +1,38 @@
+/*********************************************************
+ * Copyright (C) 1998, 2008, 2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ *  hostifMem.h - Platform dependent interface. This module defines
+ *                functions for allocating and releasing memory
+ *                from the kernel.
+ */
+
+
+#ifndef _HOSTIFMEM_H_
+#define _HOSTIFMEM_H_
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+#include "vm_basic_types.h"
+
+
+void *HostIF_AllocKernelMem(size_t size, Bool nonPageable);
+void  HostIF_FreeKernelMem(void *ptr);
+
+#endif // ifdef _HOSTIFMEM_H_

vmmon-only/common/memtrack.c

@@ -0,0 +1,563 @@
+/*********************************************************
+ * Copyright (C) 1998-2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * memtrack.c --
+ *
+ *    Utility module for tracking pinned memory, which allows later
+ *    lookup by VPN.
+ *
+ * --
+ *
+ * Track memory using a 3-level directory, to keep allocations to one
+ * page in size. The first level is inlined with the MemTrack struct
+ * and a new page is allocated for each directory in the second level,
+ * as needed. The third level packs in as many MemTrackEntry structs
+ * on a single page as possible.
+ *
+ * Optionally use a 2-level directory on systems that prefer larger
+ * contiguous allocations. In this case we allow the first level
+ * allocation to be larger than 1 page (its size will depend on the
+ * maximum number of tracked pages, currently set to 4GB).
+ *
+ *   MemTrack      MemTrackDir1        MemTrackDir2      MemTrackDir3
+ *   (Handle)                           (Optional)
+ *
+ *                                ---->/----------\
+ *                                |    | Dir[0]   |----->/----------\
+ *                                |    | Dir[1]   |      | Entry[0] |
+ *                                |    | ...      |      | Entry[1] |
+ *                                |    | Dir[N]   |      | ...      |
+ * /----------\  MEMTRACK_DIR2_ENTRIES \----------/      | Entry[N] |
+ * | ...      |                   |                      \----------/
+ * | dir1     |----/----------\   |  ->/----------\
+ * | ...      |\   | Dir[0]   |----  | | Dir[N+1] |
+ * |          | \  | Dir[1]   |------- | Dir[N+2] |
+ * \----------/  \ | ...      |        | ...      |
+ *                \| Dir[N]   |-----   | Dir[N+N] |           .
+ *                 \----------/    |   \----------/           .
+ *   MEMTRACK_DIR1_ENTRIES         |                          .
+ *                                 --->/----------\
+ *                                     | ...      |
+ *                                     |          |
+ *                                     |          |
+ *                                     | Dir[M]   |----->/----------\
+ *                                     \----------/      | ...      |
+ *                                                       |          |
+ *                                                       |          |
+ *                                                       | Entry[M] |
+ *                                                       \----------/
+ *
+ * We also keep a simple chaining hash table of entries hashed on
+ * the VPN, for quick lookup. A separate hash table hashed on the MPN
+ * exists as well, but this is only used in debug builds.
+ *
+ * This tracker does not allow pages to be removed. If, in the future,
+ * we have a use case for removing MPNs from the tracker, a simple
+ * MemTrackEntry recycle scheme can be implemented at the cost of an
+ * additional pointer per MemTrackEntry instance.
+ */
+
+#if defined(__linux__)
+/* Must come before any kernel header file. */
+#   include "driver-config.h"
+
+#   include <linux/string.h> /* memset() in the kernel */
+#elif defined(WINNT_DDK)
+#   undef PAGE_SIZE          /* Redefined in ntddk.h, and we use that defn. */
+#   undef PAGE_SHIFT
+#   include <ntddk.h>
+#else
+#   include <string.h>
+#endif
+
+#include "vmware.h"
+#include "hostif.h"
+#include "vmx86.h"
+
+#include "memtrack.h"
+
+/*
+ * Modify this value to increase the maximum number of tracked pages
+ * per MemTrack instance.
+ */
+#define MEMTRACK_MAX_TRACKED        GBYTES_2_PAGES(119)
+
+/*
+ * Linux uses a 3-level directory, because we want to keep allocations
+ * to a single page.
+ */
+#if defined(__linux__)
+#define MEMTRACK_3LEVEL             (1)
+#endif
+
+#define MEMTRACK_DIR3_ENTRIES       (PAGE_SIZE / sizeof (MemTrackEntry))
+#if defined(MEMTRACK_3LEVEL)
+#define MEMTRACK_DIR2_ENTRIES       (PAGE_SIZE / sizeof (void *))
+#else
+#define MEMTRACK_DIR2_ENTRIES       (1)
+#endif
+#define MEMTRACK_DIR1_ENTRIES       CEILING(MEMTRACK_MAX_TRACKED,        \
+                                            (MEMTRACK_DIR2_ENTRIES *     \
+                                             MEMTRACK_DIR3_ENTRIES))
+
+#define MEMTRACK_HT_SIZE            (16384)
+#define MEMTRACK_HT_ENTRIES         (PAGE_SIZE / sizeof (void *))
+#define MEMTRACK_HT_PAGES           (MEMTRACK_HT_SIZE / MEMTRACK_HT_ENTRIES)
+
+typedef struct MemTrackDir3 {
+   MemTrackEntry     entries[MEMTRACK_DIR3_ENTRIES];
+} MemTrackDir3;
+
+#if defined(MEMTRACK_3LEVEL)
+typedef struct MemTrackDir2 {
+   MemTrackDir3     *dir[MEMTRACK_DIR2_ENTRIES];
+} MemTrackDir2;
+#else
+typedef struct MemTrackDir3 MemTrackDir2;
+#endif
+
+typedef struct MemTrackDir1 {
+   MemTrackDir2     *dir[MEMTRACK_DIR1_ENTRIES];
+} MemTrackDir1;
+
+typedef struct MemTrackHTPage {
+   MemTrackEntry    *entries[MEMTRACK_HT_ENTRIES];
+} MemTrackHTPage;
+
+typedef struct MemTrackHT {
+   MemTrackHTPage   *pages[MEMTRACK_HT_PAGES];
+} MemTrackHT;
+
+typedef uint64 MemTrackHTKey;
+
+typedef struct MemTrack {
+   VMDriver         *vm;            /* The VM instance. */
+   PageCnt           numPages;      /* Number of pages tracked. */
+   MemTrackDir1      dir1;          /* First level directory. */
+   MemTrackHT        vpnHashTable;  /* VPN to entry hashtable. */
+   MemTrackHT       *mpnHashTable;  /* MPN to entry hashtable. */
+} MemTrack;
+
+/*
+ * The following functions and macros help allocate and access the
+ * directory structure. This is convenient because the second level
+ * directory is optional.
+ */
+
+#define MEMTRACK_IDX2DIR(_idx, _p1, _p2, _p3)                          \
+   do {                                                                \
+      _p1   = _idx / (MEMTRACK_DIR2_ENTRIES * MEMTRACK_DIR3_ENTRIES);  \
+      _p2   = (_idx / MEMTRACK_DIR3_ENTRIES) % MEMTRACK_DIR2_ENTRIES;  \
+      _p3   = _idx % MEMTRACK_DIR3_ENTRIES;                            \
+   } while (0)
+
+#define MEMTRACK_GETDIR2(_dir1, _p1)     (_dir1->dir[_p1])
+#define MEMTRACK_ALLOCDIR2(_dir1, _p1)   MemTrackAllocDir2(_dir1, _p1)
+#define MEMTRACK_FREEDIR2(_dir1)         HostIF_FreePage(_dir1)
+
+#if defined(MEMTRACK_3LEVEL)
+#define MEMTRACK_GETENTRY(_dir1, _p1, _p2, _p3) \
+                                         (&((_dir1->dir[_p1])->dir[_p2])->entries[_p3])
+#define MEMTRACK_GETDIR3(_dir2, _p2)     (_dir2->dir[_p2])
+#define MEMTRACK_ALLOCDIR3(_dir2, _p2)   MemTrackAllocDir3(_dir2, _p2)
+#define MEMTRACK_FREEDIR3(_dir2)         HostIF_FreePage(_dir2)
+#else
+#define MEMTRACK_GETENTRY(_dir1, _p1, _p2, _p3) \
+                                         (&(_dir1->dir[_p1])->entries[_p3])
+#define MEMTRACK_GETDIR3(_dir2, _p2)     (_dir2)
+#define MEMTRACK_ALLOCDIR3(_dir2, _p2)   (_dir2)
+#define MEMTRACK_FREEDIR3(_dir2)
+#endif
+
+static INLINE void *
+MemTrackAllocPage(void)
+{
+   void *ptr = HostIF_AllocPage();
+   if (ptr != NULL) {
+      memset(ptr, 0, PAGE_SIZE);
+   }
+   return ptr;
+}
+
+#define MEMTRACK_ALLOCDFN(_name, _itype, _otype)   \
+   static INLINE _otype *                          \
+   _name(_itype *arg, uint64 pos)                  \
+   {                                               \
+      if (arg->dir[pos] == NULL) {                 \
+         arg->dir[pos] = MemTrackAllocPage();      \
+      }                                            \
+      return arg->dir[pos];                        \
+   }
+
+#if defined(MEMTRACK_3LEVEL)
+MEMTRACK_ALLOCDFN(MemTrackAllocDir3, MemTrackDir2, MemTrackDir3)
+#endif
+MEMTRACK_ALLOCDFN(MemTrackAllocDir2, MemTrackDir1, MemTrackDir2)
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MemTrackHTLookup --
+ * MemTrackHTInsert --
+ *
+ *      Helper functions to insert or lookup entries in the VPN or
+ *      MPN hash tables. Hash tables are always allocated in page
+ *      size chunks.
+ *
+ *----------------------------------------------------------------------
+ */
+
+#define MEMTRACK_HASHKEY(_key, _hash, _page, _pos)  \
+   do {                                             \
+      _hash = _key % MEMTRACK_HT_SIZE;              \
+      _page = _hash / MEMTRACK_HT_ENTRIES;          \
+      _pos  = _hash % MEMTRACK_HT_ENTRIES;          \
+   } while(0)
+
+static INLINE MemTrackEntry **
+MemTrackHTLookup(MemTrackHT *ht,       // IN
+                 MemTrackHTKey key)    // IN
+{
+   uint64 hash, page, pos;
+
+   MEMTRACK_HASHKEY(key, hash, page, pos);
+
+   return &ht->pages[page]->entries[pos];
+}
+
+static INLINE void
+MemTrackHTInsert(MemTrackHT *ht,          // IN
+                 MemTrackEntry *ent,      // IN
+                 MemTrackEntry **chain,   // OUT
+                 MemTrackHTKey key)       // IN
+{
+   MemTrackEntry **head = MemTrackHTLookup(ht, key);
+   *chain = *head;
+   *head = ent;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MemTrackCleanup --
+ *
+ *      Deallocate all memory associated with the specified tracker.
+ *
+ * Results:
+ *      None.
+ *
+ * Side effects:
+ *      Memory deallocation.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static void
+MemTrackCleanup(MemTrack *mt)    // IN
+{
+   PageCnt idx;
+   uint64 p1;
+   MemTrackDir1 *dir1;
+   Bool freeBackMap = mt != NULL && mt->mpnHashTable != NULL;
+
+   if (mt == NULL) {
+      return;
+   }
+   dir1 = &mt->dir1;
+
+   for (p1 = 0; p1 < MEMTRACK_DIR1_ENTRIES; p1++) {
+      uint64 p2;
+      MemTrackDir2 *dir2 = MEMTRACK_GETDIR2(dir1, p1);
+
+      if (dir2 == NULL) {
+         break;
+      }
+      for (p2 = 0; p2 < MEMTRACK_DIR2_ENTRIES; p2++) {
+         MemTrackDir3 *dir3 = MEMTRACK_GETDIR3(dir2, p2);
+
+         if (dir3 == NULL) {
+            break;
+         }
+         MEMTRACK_FREEDIR3(dir3);
+      }
+      MEMTRACK_FREEDIR2(dir2);
+   }
+
+   for (idx = 0; idx < MEMTRACK_HT_PAGES; idx++) {
+      if (mt->vpnHashTable.pages[idx] != NULL) {
+         HostIF_FreePage(mt->vpnHashTable.pages[idx]);
+      }
+      if (freeBackMap) {
+         if (mt->mpnHashTable->pages[idx] != NULL) {
+            HostIF_FreePage(mt->mpnHashTable->pages[idx]);
+         }
+      }
+   }
+   if (freeBackMap) {
+      HostIF_FreeKernelMem(mt->mpnHashTable);
+   }
+
+   HostIF_FreeKernelMem(mt);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MemTrack_Init --
+ *
+ *      Allocate and initialize the tracker.
+ *
+ * Results:
+ *      Handle used to access the tracker.
+ *
+ * Side effects:
+ *      Memory allocation.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MemTrack *
+MemTrack_Init(VMDriver *vm) // IN:
+{
+   MemTrack *mt;
+   PageCnt idx;
+
+#if defined(MEMTRACK_3LEVEL)
+   ASSERT_ON_COMPILE(sizeof *mt <= PAGE_SIZE);
+   ASSERT_ON_COMPILE(sizeof (MemTrackDir2) == PAGE_SIZE);
+#endif
+   ASSERT_ON_COMPILE(sizeof (MemTrackDir3) <= PAGE_SIZE);
+
+   mt = HostIF_AllocKernelMem(sizeof *mt, FALSE);
+   if (mt == NULL) {
+      Warning("MemTrack failed to allocate handle.\n");
+      goto error;
+   }
+   memset(mt, 0, sizeof *mt);
+   mt->vm = vm;
+
+   for (idx = 0; idx < MEMTRACK_HT_PAGES; idx++) {
+      MemTrackHTPage *htPage = MemTrackAllocPage();
+
+      if (htPage == NULL) {
+         Warning("MemTrack failed to allocate VPN hash table (%"FMT64"d).\n",
+                 idx);
+         goto error;
+      }
+      mt->vpnHashTable.pages[idx] = htPage;
+   }
+
+   mt->mpnHashTable = HostIF_AllocKernelMem(sizeof *mt->mpnHashTable, FALSE);
+   if (mt->mpnHashTable == NULL) {
+      Warning("MemTrack failed to allocate MPN hash table.\n");
+      goto error;
+   }
+   memset(mt->mpnHashTable, 0, sizeof *mt->mpnHashTable);
+   for (idx = 0; idx < MEMTRACK_HT_PAGES; idx++) {
+      MemTrackHTPage *htPage = MemTrackAllocPage();
+      if (htPage == NULL) {
+         Warning("MemTrack failed to allocate MPN hash table (%"FMT64"d).\n",
+                 idx);
+         goto error;
+      }
+      mt->mpnHashTable->pages[idx] = htPage;
+   }
+
+   return mt;
+
+error:
+   MemTrackCleanup(mt);
+   return NULL;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MemTrack_Add --
+ *
+ *      Add the specified VPN:MPN pair to the memory tracker.
+ *
+ * Results:
+ *      A pointer to the element, or NULL on error.
+ *
+ * Side effects:
+ *      Memory allocation.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MemTrackEntry *
+MemTrack_Add(MemTrack *mt,    // IN
+             VPN64 vpn,       // IN
+             MPN mpn)         // IN
+{
+   PageCnt idx = mt->numPages;
+   uint64 p1, p2, p3;
+   MemTrackEntry *ent;
+   MemTrackDir1 *dir1 = &mt->dir1;
+   MemTrackDir2 *dir2;
+   MemTrackDir3 *dir3;
+   MEMTRACK_IDX2DIR(idx, p1, p2, p3);
+
+
+   ASSERT(HostIF_VMLockIsHeld(mt->vm));
+
+   if (p1 >= MEMTRACK_DIR1_ENTRIES ||
+       p2 >= MEMTRACK_DIR2_ENTRIES ||
+       p3 >= MEMTRACK_DIR3_ENTRIES) {
+      return NULL;
+   }
+
+   dir2 = MEMTRACK_ALLOCDIR2(dir1, p1);
+   if (dir2 == NULL) {
+      return NULL;
+   }
+
+   dir3 = MEMTRACK_ALLOCDIR3(dir2, p2);
+   if (dir3 == NULL) {
+      return NULL;
+   }
+
+   ent = MEMTRACK_GETENTRY(dir1, p1, p2, p3);
+   ent->vpn = vpn;
+   ent->mpn = mpn;
+
+   MemTrackHTInsert(&mt->vpnHashTable, ent, &ent->vpnChain, ent->vpn);
+   MemTrackHTInsert(mt->mpnHashTable, ent, &ent->mpnChain, ent->mpn);
+
+   mt->numPages++;
+
+   return ent;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MemTrack_LookupVPN --
+ *
+ *      Lookup the specified VPN address in the memory tracker.
+ *
+ * Results:
+ *      A pointer to the element, or NULL if not there.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MemTrackEntry *
+MemTrack_LookupVPN(MemTrack *mt, // IN
+                   VPN64 vpn)    // IN
+{
+   MemTrackEntry *next = *MemTrackHTLookup(&mt->vpnHashTable, vpn);
+   ASSERT(HostIF_VMLockIsHeld(mt->vm));
+
+   while (next != NULL) {
+      if (next->vpn == vpn) {
+         return next;
+      }
+      next = next->vpnChain;
+   }
+
+   return NULL;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MemTrack_LookupMPN --
+ *
+ *      Lookup the specified MPN address in the memory tracker.
+ *
+ * Results:
+ *      A pointer to the element, or NULL if not there.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+MemTrackEntry *
+MemTrack_LookupMPN(MemTrack *mt, // IN
+                   MPN mpn)      // IN
+{
+   MemTrackEntry *next;
+   ASSERT(HostIF_VMLockIsHeld(mt->vm));
+   next = *MemTrackHTLookup(mt->mpnHashTable, mpn);
+
+   while (next != NULL) {
+      if (next->mpn == mpn) {
+         return next;
+      }
+      next = next->mpnChain;
+   }
+
+   return NULL;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MemTrack_Cleanup --
+ *
+ *      Cleanup all resources allocated for the tracker. For
+ *      all pages in the tracker call the user provided clean up
+ *      function.
+ *
+ * Results:
+ *      Number of pages in the tracker.
+ *
+ * Side effects:
+ *      Memory deallocation.
+ *
+ *----------------------------------------------------------------------
+ */
+
+PageCnt
+MemTrack_Cleanup(MemTrack *mt,            // IN
+                 MemTrackCleanupCb *cb,   // IN
+                 void *cData)             // IN
+{
+   PageCnt idx;
+   PageCnt count = 0;
+
+   for (idx = 0; idx < mt->numPages; idx++) {
+      uint64 p1, p2, p3;
+      MemTrackEntry *ent;
+      MemTrackDir1 *dir1 = &mt->dir1;
+      MEMTRACK_IDX2DIR(idx, p1, p2, p3);
+
+      ent = MEMTRACK_GETENTRY(dir1, p1, p2, p3);
+      cb(cData, ent);
+
+      count++;
+   }
+
+   MemTrackCleanup(mt);
+
+   return count;
+}

vmmon-only/common/memtrack.h

@@ -0,0 +1,52 @@
+/*********************************************************
+ * Copyright (C) 1998,2017-2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * memtrack.h --
+ *
+ *    Utility module for tracking pinned memory, which allows later
+ *    lookup by VPN and optionally by MPN.
+ */
+
+
+#ifndef _MEMTRACK_H_
+#define _MEMTRACK_H_
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+struct MemTrack;
+
+typedef struct MemTrackEntry {
+   VPN64                   vpn;
+   MPN                     mpn;
+   struct MemTrackEntry   *vpnChain;
+   struct MemTrackEntry   *mpnChain;
+} MemTrackEntry;
+
+typedef void (MemTrackCleanupCb)(void *cData, MemTrackEntry *entry);
+
+extern struct MemTrack *MemTrack_Init(VMDriver *vm);
+extern PageCnt MemTrack_Cleanup(struct MemTrack *mt, MemTrackCleanupCb *cb,
+                                void *cbData);
+extern MemTrackEntry *MemTrack_Add(struct MemTrack *mt, VPN64 vpn, MPN mpn);
+extern MemTrackEntry *MemTrack_LookupVPN(struct MemTrack *mt, VPN64 vpn);
+extern MemTrackEntry *MemTrack_LookupMPN(struct MemTrack *mt, MPN mpn);
+
+#endif // _MEMTRACK_H_

vmmon-only/common/phystrack.c

@@ -0,0 +1,521 @@
+/*********************************************************
+ * Copyright (C) 1998,2014,2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * phystrack.c --
+ *
+ *    track down the utilization of the physical pages.
+ *
+ *    Depending on configuration phystracker provides either 2-level or
+ *    3-level structure to track whether page (specified by its MPN) is
+ *    locked or no.  Linux uses 3-level structures with top limit of
+ *    1TB (32bit) or 16TB (64bit).  Windows and Mac use 2-level structures
+ *    ready to hold 128GB (32bit) or 2TB (64bit) of memory.
+ *
+ *    2-level phystracker is built on top of 3-level one by collapsing
+ *    middle level.
+ */
+
+
+#ifdef __linux__
+/* Must come before any kernel header file --hpreg */
+#   include "driver-config.h"
+
+#   include <linux/string.h> /* memset() in the kernel */
+#else
+#   include <string.h>
+#endif
+
+#include "vmware.h"
+#include "vmx86.h"
+#include "phystrack.h"
+#include "hostif.h"
+
+#define BYTES_PER_ENTRY      (PAGE_SIZE)
+#define PHYSTRACK_L3_ENTRIES (8 * BYTES_PER_ENTRY) /* 128MB */
+
+#if defined(__linux__)
+#define PHYSTRACK_L2_ENTRIES (BYTES_PER_ENTRY / sizeof(void *)) /* 64GB or 128GB */
+/*
+ * Currently MPN is 32 bits.  15 bits are in L3, 9 bits are in L2,
+ * leaving 8 bits for L1...
+ */
+#define PHYSTRACK_L1_ENTRIES (256) /* 16TB. */
+#else
+#define PHYSTRACK_L1_ENTRIES (PHYSTRACK_MAX_SUPPORTED_GB * 8)
+#endif
+
+#ifndef PHYSTRACK_L2_ENTRIES
+#define PHYSTRACK_L2_ENTRIES (1)
+#else
+#define PHYSTRACK_3LEVEL (1)
+#endif
+
+typedef struct PhysTrackerL3 {
+   uint8 bits[BYTES_PER_ENTRY];
+} PhysTrackerL3;
+
+#ifdef PHYSTRACK_3LEVEL
+typedef struct PhysTrackerL2 {
+   PhysTrackerL3 *dir[PHYSTRACK_L2_ENTRIES];
+} PhysTrackerL2;
+#else
+typedef struct PhysTrackerL3 PhysTrackerL2;
+#endif
+
+typedef struct PhysTracker {
+   VMDriver *vm; /* Used only for debugging and asserts. */
+   PhysTrackerL2 *dir[PHYSTRACK_L1_ENTRIES];
+} PhysTracker;
+
+
+/*
+ * Convert MPN to p1, p2, and p3 indices.  p1/p2/p3 must be l-values.
+ * Currently we support a 64 bit container for an MPN
+ * in hosted but not an actual 64 bit value as no hosted OS
+ * supports this yet. Hence in PhysMem tracker we are deliberately using
+ * a 32-bit container to save memory. Also the tracker is allocating pages
+ * considering the MPN to be a 32 bit value. This will change once we get
+ * systems supporting 64 bit memory/addressing space.
+ * Until then let us assert if a value greater than 32 bit is being passed.
+ */
+#define PHYSTRACK_MPN2IDX(mpn, p1, p2, p3)           \
+   do {                                              \
+      ASSERT((mpn >> 32) == 0);                      \
+      p2 = (unsigned)(mpn) / PHYSTRACK_L3_ENTRIES;   \
+      p1 = p2 / PHYSTRACK_L2_ENTRIES;                \
+      p2 = p2 % PHYSTRACK_L2_ENTRIES;                \
+      p3 = (unsigned)(mpn) % PHYSTRACK_L3_ENTRIES;   \
+   } while (0)
+
+/*
+ * Convert L3 index to offset and bitmask.  offs/bitmask must be l-values.
+ */
+#define PHYSTRACK_GETL3POS(p3, offs, bitmask) \
+   do {                          \
+      offs = (p3) / 8;           \
+      bitmask = 1 << ((p3) % 8); \
+   } while (0)
+
+/*
+ * Helpers hiding middle level.
+ */
+#ifdef PHYSTRACK_3LEVEL
+#define PHYSTRACK_GETL3(dir2, p2) (dir2)->dir[(p2)]
+#define PHYSTRACK_ALLOCL3(dir2, p2) PhysTrackAllocL3((dir2), (p2))
+#define PHYSTRACK_FREEL3(dir2, p2) PhysTrackFreeL3((dir2), (p2))
+#else
+#define PHYSTRACK_GETL3(dir2, p2) (dir2)
+#define PHYSTRACK_ALLOCL3(dir2, p2) (dir2)
+#define PHYSTRACK_FREEL3(dir2, p2) do { } while (0)
+#endif
+
+
+#ifdef PHYSTRACK_3LEVEL
+/*
+ *----------------------------------------------------------------------
+ *
+ * PhysTrackAllocL3 --
+ *
+ *      Allocate and hook L3 table to the L2 directory if does not exist.
+ *      Or get existing one if it exists.
+ *
+ * Results:
+ *      L3 table.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE_SINGLE_CALLER PhysTrackerL3 *
+PhysTrackAllocL3(PhysTrackerL2 *dir2,
+                 unsigned int p2)
+{
+   PhysTrackerL3 *dir3;
+
+   dir3 = dir2->dir[p2];
+   if (!dir3) {
+      ASSERT_ON_COMPILE(sizeof *dir3 == PAGE_SIZE);
+      dir3 = HostIF_AllocPage();
+      if (!dir3) {
+         PANIC();
+      }
+      memset(dir3, 0, sizeof *dir3);
+      dir2->dir[p2] = dir3;
+   }
+   return dir3;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * PhysTrackFreeL3 --
+ *
+ *      Unhook L3 table from L2 directory, and free it.
+ *
+ * Results:
+ *      None.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE_SINGLE_CALLER void
+PhysTrackFreeL3(PhysTrackerL2 *dir2,
+                unsigned int p2)
+{
+   HostIF_FreePage(dir2->dir[p2]);
+   dir2->dir[p2] = NULL;
+}
+#endif
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * PhysTrack_Alloc --
+ *
+ *      Create new PhysTracker.
+ *
+ * Results:
+ *      Creates new PhysTracker.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+PhysTracker *
+PhysTrack_Alloc(VMDriver *vm)
+{
+   PhysTracker *tracker;
+
+   ASSERT(vm != NULL);
+
+   /* allocate a new phystracker */
+   tracker = HostIF_AllocKernelMem(sizeof *tracker, FALSE);
+   if (tracker) {
+      memset(tracker, 0, sizeof *tracker);
+      tracker->vm = vm;
+   } else {
+      Warning("%s failed\n", __FUNCTION__);
+   }
+
+   return tracker;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * PhysTrack_Free --
+ *
+ *      module deallocation
+ *
+ * Results:
+ *      reallocates all structures, including 'tracker'
+ *
+ * Side effects:
+ *      tracker deallocated
+ *
+ *----------------------------------------------------------------------
+ */
+
+void
+PhysTrack_Free(PhysTracker *tracker)
+{
+   unsigned int p1;
+
+   ASSERT(tracker);
+
+   for (p1 = 0; p1 < PHYSTRACK_L1_ENTRIES; p1++) {
+      PhysTrackerL2 *dir2 = tracker->dir[p1];
+
+      if (dir2) {
+         unsigned int p2;
+
+         for (p2 = 0; p2 < PHYSTRACK_L2_ENTRIES; p2++) {
+            PhysTrackerL3 *dir3 = PHYSTRACK_GETL3(dir2, p2);
+
+            if (dir3) {
+               unsigned int pos;
+
+               for (pos = 0; pos < BYTES_PER_ENTRY; pos++) {
+                  if (dir3->bits[pos]) {
+                     Warning("%s: pfns still locked\n", __FUNCTION__);
+                     PANIC();
+                  }
+               }
+               PHYSTRACK_FREEL3(dir2, p2);
+            }
+         }
+         HostIF_FreePage(dir2);
+         tracker->dir[p1] = NULL;
+      }
+   }
+   HostIF_FreeKernelMem(tracker);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * PhysTrack_Add --
+ *
+ *      add a page to the core map tracking.
+ *
+ * Results:
+ *      void
+ *
+ * Side effects:
+ *      Fatal if the page is already tracked.
+ *
+ *----------------------------------------------------------------------
+ */
+
+void
+PhysTrack_Add(PhysTracker *tracker, // IN/OUT
+              MPN mpn)              // IN: MPN of page to be added
+{
+   unsigned int p1;
+   unsigned int p2;
+   unsigned int p3;
+   unsigned int pos;
+   unsigned int bit;
+   PhysTrackerL2 *dir2;
+   PhysTrackerL3 *dir3;
+
+   ASSERT(tracker);
+   ASSERT(HostIF_VMLockIsHeld(tracker->vm));
+   PHYSTRACK_MPN2IDX(mpn, p1, p2, p3);
+   ASSERT(p1 < PHYSTRACK_L1_ENTRIES);
+
+   dir2 = tracker->dir[p1];
+   if (!dir2) {
+      // more efficient with page alloc
+      ASSERT_ON_COMPILE(sizeof *dir2 == PAGE_SIZE);
+      dir2 = HostIF_AllocPage();
+      if (!dir2) {
+         PANIC();
+      }
+      memset(dir2, 0, sizeof *dir2);
+      tracker->dir[p1] = dir2;
+   }
+   dir3 = PHYSTRACK_ALLOCL3(dir2, p2);
+   PHYSTRACK_GETL3POS(p3, pos, bit);
+   if (dir3->bits[pos] & bit) {
+      PANIC();
+   }
+   dir3->bits[pos] |= bit;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * PhysTrack_Remove --
+ *
+ *      remove a page from the core map tracking
+ *
+ * Results:
+ *      void
+ *
+ * Side effects:
+ *      Fatal if the page is not tracked
+ *
+ *----------------------------------------------------------------------
+ */
+
+void
+PhysTrack_Remove(PhysTracker *tracker, // IN/OUT
+                 MPN mpn)              // IN: MPN of page to be removed.
+{
+   unsigned int p1;
+   unsigned int p2;
+   unsigned int p3;
+   unsigned int pos;
+   unsigned int bit;
+   PhysTrackerL2 *dir2;
+   PhysTrackerL3 *dir3;
+
+   ASSERT(tracker);
+   ASSERT(HostIF_VMLockIsHeld(tracker->vm));
+   PHYSTRACK_MPN2IDX(mpn, p1, p2, p3);
+   ASSERT(p1 < PHYSTRACK_L1_ENTRIES);
+
+   dir2 = tracker->dir[p1];
+   if (!dir2) {
+      PANIC();
+   }
+   dir3 = PHYSTRACK_GETL3(dir2, p2);
+   if (!dir3) {
+      PANIC();
+   }
+   PHYSTRACK_GETL3POS(p3, pos, bit);
+   if (!(dir3->bits[pos] & bit)) {
+      PANIC();
+   }
+   dir3->bits[pos] &= ~bit;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * PhysTrack_Test --
+ *
+ *      tests whether a page is being tracked
+ *
+ * Results:
+ *      TRUE if the page is tracked
+ *      FALSE otherwise
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+Bool
+PhysTrack_Test(const PhysTracker *tracker, // IN
+               MPN mpn)                    // IN: MPN of page to be tested.
+{
+   unsigned int p1;
+   unsigned int p2;
+   unsigned int p3;
+   unsigned int pos;
+   unsigned int bit;
+   PhysTrackerL2 *dir2;
+   PhysTrackerL3 *dir3;
+
+   ASSERT(tracker);
+   ASSERT(HostIF_VMLockIsHeld(tracker->vm));
+   PHYSTRACK_MPN2IDX(mpn, p1, p2, p3);
+   if (p1 >= PHYSTRACK_L1_ENTRIES) {
+      return FALSE;
+   }
+   dir2 = tracker->dir[p1];
+   if (!dir2) {
+      return FALSE;
+   }
+   dir3 = PHYSTRACK_GETL3(dir2, p2);
+   if (!dir3) {
+      return FALSE;
+   }
+   PHYSTRACK_GETL3POS(p3, pos, bit);
+   return (dir3->bits[pos] & bit) != 0;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * PhysTrack_GetNext --
+ *
+ *      Return next tracked page
+ *
+ * Results:
+ *      MPN when some tracked page was found
+ *      INVALID_MPN otherwise.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MPN
+PhysTrack_GetNext(const PhysTracker *tracker, // IN
+                  MPN mpn)                    // IN: MPN of page to be tracked.
+{
+   unsigned int p1;
+   unsigned int p2;
+   unsigned int p3;
+
+   if (mpn == INVALID_MPN) {
+      mpn = 0; /* First iteration. */
+   } else {
+      mpn++;   /* We want the next MPN. */
+   }
+   PHYSTRACK_MPN2IDX(mpn, p1, p2, p3);
+
+   ASSERT(tracker);
+   ASSERT(HostIF_VMLockIsHeld(tracker->vm));
+   for (; p1 < PHYSTRACK_L1_ENTRIES; p1++) {
+      PhysTrackerL2 *dir2;
+
+      dir2 = tracker->dir[p1];
+      if (dir2) {
+         for (; p2 < PHYSTRACK_L2_ENTRIES; p2++) {
+            PhysTrackerL3 *dir3;
+
+            dir3 = PHYSTRACK_GETL3(dir2, p2);
+            if (dir3) {
+               for (; p3 < PHYSTRACK_L3_ENTRIES; p3++) {
+                  unsigned int pos;
+                  unsigned int bit;
+
+                  PHYSTRACK_GETL3POS(p3, pos, bit);
+                  if (dir3->bits[pos] & bit) {
+                     return (p1 * PHYSTRACK_L2_ENTRIES + p2) * PHYSTRACK_L3_ENTRIES + p3;
+                  }
+               }
+            }
+            p3 = 0;
+         }
+      }
+      p2 = 0; p3 = 0;
+   }
+   return INVALID_MPN;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * PhysTrack_GetNumTrackedPages --
+ *
+ *      Returns the total number of tracked pages
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+PageCnt
+PhysTrack_GetNumTrackedPages(const PhysTracker *tracker)
+{
+   PageCnt numTrackedMPNs = 0;
+   MPN nextMPN;
+   ASSERT(tracker);
+   ASSERT(HostIF_VMLockIsHeld(tracker->vm));
+   nextMPN = PhysTrack_GetNext(tracker, INVALID_MPN);
+   while (nextMPN != INVALID_MPN) {
+      numTrackedMPNs++;
+      nextMPN = PhysTrack_GetNext(tracker, nextMPN);
+   }
+   return numTrackedMPNs;
+}
+

vmmon-only/common/phystrack.h

@@ -0,0 +1,55 @@
+/*********************************************************
+ * Copyright (C) 1998,2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * phystrack.h --
+ *
+ *    track down the utilization of the physical pages
+ */
+
+#ifndef PHYSTRACK_H
+#define PHYSTRACK_H
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#if !defined (__linux__)
+/* XXX: can be more efficient based on server vs. desktop and version of Windows */
+#define PHYSTRACK_MAX_SUPPORTED_GB (2048 + 4) /* 2 TB 64-bit W2k8 + 4 GB PCI */
+#endif
+
+struct PhysTracker;
+struct VMDriver;
+
+EXTERN struct PhysTracker *PhysTrack_Alloc(struct VMDriver *vm);
+EXTERN void PhysTrack_Free(struct PhysTracker *);
+
+EXTERN void PhysTrack_Add(struct PhysTracker *, MPN);
+EXTERN void PhysTrack_Remove(struct PhysTracker *, MPN);
+EXTERN Bool PhysTrack_Test(const struct PhysTracker *, MPN);
+EXTERN MPN PhysTrack_GetNext(const struct PhysTracker *, MPN);
+EXTERN PageCnt PhysTrack_GetNumTrackedPages(const struct PhysTracker *);
+
+#endif
+
+
+
+
+
+

vmmon-only/common/sharedAreaVmmon.c

@@ -0,0 +1,299 @@
+/*********************************************************
+ * Copyright (C) 2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * sharedAreaVmmon.c --
+ *
+ *     VMMon shared area management.
+ */
+
+#ifdef __linux__
+#   include "driver-config.h"
+#   include <linux/string.h> /* memset() in the kernel. */
+#else
+#   include <string.h>
+#endif
+#include "vmware.h"
+#include "vm_assert.h"
+#include "hostif.h"
+#include "sharedAreaVmmon.h"
+#include "vmx86.h"
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * SharedAreaVmmonIsMultiVCPU --
+ *
+ *      Helper function for determining if a shared area type represents
+ *      a region that has VCPU-specific memory.
+ *
+ * Results:
+ *      TRUE if type has VCPU-specific memory, FALSE otherwise.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static Bool
+SharedAreaVmmonIsMultiVCPU(SharedAreaType type)
+{
+   switch (type) {
+   case SHARED_AREA_PER_VM:
+   case SHARED_AREA_PER_VM_VMX:
+      return FALSE;
+   default:
+      return TRUE;
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * SharedAreaVmmon_Init --
+ *
+ *      Initializes the shared area component of the VM Driver.
+ *
+ * Results:
+ *      A pointer to the VMMon shared area.
+ *
+ *----------------------------------------------------------------------
+ */
+
+SharedAreaVmmon *
+SharedAreaVmmon_Init(VMDriver *vm)
+{
+   SharedAreaVmmon *sa;
+
+   /*
+    * Allocate the shared area pointer here. The pages in each region are
+    * lazily allocated when a region is registered.
+    */
+   sa = HostIF_AllocKernelMem(sizeof *sa, FALSE);
+   if (sa == NULL) {
+      Warning("SharedArea failed to allocate handle.\n");
+      return NULL;
+   }
+   memset(sa, 0, sizeof *sa);
+   return sa;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * SharedAreaVmmon_Cleanup --
+ *
+ *      Cleans up the shared area component of the VM Driver by freeing all
+ *      previously allocated VMMon shared area memory.
+ *
+ * Results:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+void
+SharedAreaVmmon_Cleanup(SharedAreaVmmon *area)
+{
+   SharedAreaType t;
+
+   if (area == NULL) {
+      return;
+   }
+
+   for (t = 0; t < NUM_SHARED_AREAS; t++) {
+      SharedAreaVmmonRegion *region = &area->regions[t];
+
+      if (region->pages != NULL) {
+         HostIF_FreeKernelMem(region->pages);
+      }
+   }
+   HostIF_FreeKernelMem(area);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * SharedAreaVmmon_ValidateRegionArgs --
+ *
+ *      Validate the given VMMon shared area region registration block.
+ *
+ * Results:
+ *      TRUE if the block is valid and can be used to register a shared
+ *      area region for a VCPU, FALSE otherwise.
+ *
+ *----------------------------------------------------------------------
+ */
+
+Bool
+SharedAreaVmmon_ValidateRegionArgs(VMDriver *vm,
+                                   VMSharedAreaRegistrationBlock *block)
+{
+   SharedAreaVmmonRegion *region;
+   Vcpuid vcpu         = block->vcpu;
+   SharedAreaType type = block->region.index;
+   PageCnt numPages    = block->region.numPages;
+
+   if (vm == NULL               ||
+       vm->sharedArea == NULL   ||
+       vcpu >= vm->numVCPUs     ||
+       numPages == 0            ||
+       type >= NUM_SHARED_AREAS ||
+       (!SharedAreaVmmonIsMultiVCPU(type) && vcpu > 0)) {
+      return FALSE;
+   }
+
+   region = &vm->sharedArea->regions[type];
+   if (region->pagesPerVcpu != 0) {
+
+      /*
+       * A region that was previously registered for a given VCPU should have
+       * the same number of pages as originally specified. Also, a region can
+       * only be reserved once for a given VCPU throughout the runtime of a VM.
+       */
+      return region->pagesPerVcpu == numPages &&
+             region->pages[vcpu * region->pagesPerVcpu] == INVALID_MPN;
+   } else {
+      /* First registration for this region. */
+      ASSERT(region->pages == NULL);
+      return TRUE;
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * SharedAreaVmmon_RegisterRegion --
+ *
+ *      Register a VMMon shared area region for a given VCPU with the VM Driver.
+ *      A successful return value indicates that the backing pages of the
+ *      shared area region are locked and tracked by the VM Driver.
+ *
+ * Results:
+ *      TRUE if the region was successfully registered for the given VCPU,
+ *      FALSE otherwise.
+ *
+ *----------------------------------------------------------------------
+ */
+
+Bool
+SharedAreaVmmon_RegisterRegion(VMDriver *vm,
+                               VMSharedAreaRegistrationBlock *block)
+{
+   Vcpuid vcpu = block->vcpu;
+   SharedAreaType type = block->region.index;
+   SharedAreaVmmonRegion *region;
+   MPN *pages;
+   PageCnt page;
+
+   ASSERT(SharedAreaVmmon_ValidateRegionArgs(vm, block));
+   region = &vm->sharedArea->regions[type];
+   if (region->pagesPerVcpu == 0) {
+      PageCnt pagesInRegion = block->region.numPages;
+
+      if (SharedAreaVmmonIsMultiVCPU(type)) {
+         pagesInRegion *= vm->numVCPUs;
+      }
+      region->pages = HostIF_AllocKernelMem(sizeof *pages * pagesInRegion,
+                                            FALSE);
+      if (region->pages == NULL) {
+         Warning("Failed to allocate pages array for region %u\n", type);
+         return FALSE;
+      }
+      for (page = 0; page < pagesInRegion; page++) {
+         region->pages[page] = INVALID_MPN;
+      }
+      region->pagesPerVcpu = block->region.numPages;
+   }
+
+   pages = region->pages + vcpu * region->pagesPerVcpu;
+   for (page = 0; page < region->pagesPerVcpu; page++) {
+      ASSERT(pages[page] == INVALID_MPN);
+   }
+
+   /*
+    * Lock all shared area backing pages throughout the runtime of the monitor.
+    * The pages remain locked until they are freed by HostIF during VM Driver
+    * tear-down.
+    */
+   for (page = 0; page < region->pagesPerVcpu; page++) {
+      VPN vmxVPN = block->region.baseVpn + page;
+      VA64 uAddr = VPN_2_VA(vmxVPN);
+      int status = Vmx86_LockPage(vm, uAddr, FALSE, pages + page);
+
+      if (status != PAGE_LOCK_SUCCESS) {
+         PageCnt resetPage;
+
+         /* Reset the region pages for this VCPU to their original state. */
+         for (resetPage = 0; resetPage < page; resetPage++) {
+            status = Vmx86_UnlockPage(vm, uAddr);
+            ASSERT(status == PAGE_UNLOCK_SUCCESS);
+            pages[resetPage] = INVALID_MPN;
+         }
+         return FALSE;
+      }
+   }
+
+   return TRUE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * SharedAreaVmmon_GetRegionMPN --
+ *
+ *      For the shared area region corresponding to the given type and VCPU,
+ *      get the MPN backing the region at the given offset.
+ *
+ * Results:
+ *      The MPN backing the region, or INVALID_MPN if no MPN is found.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MPN
+SharedAreaVmmon_GetRegionMPN(VMDriver *vm, SharedAreaVmmonRequest *request)
+{
+   SharedAreaType type = request->type;
+   Vcpuid vcpu = request->vcpu;
+   SharedAreaVmmonRegion *region;
+   PageCnt pgOffset = request->offset;
+   MPN *pages;
+
+   ASSERT(vcpu < vm->numVCPUs);
+   ASSERT(type < NUM_SHARED_AREAS);
+   ASSERT(IMPLIES(!SharedAreaVmmonIsMultiVCPU(type), vcpu == 0));
+   region = &vm->sharedArea->regions[type];
+   pages = region->pages + vcpu * region->pagesPerVcpu;
+   if (region->pages == NULL || region->pagesPerVcpu == 0) {
+      Warning("(%s) Requested unregistered region %u, VCPU %u\n", __FUNCTION__,
+              type, vcpu);
+      return INVALID_MPN;
+   }
+   if (pgOffset >= region->pagesPerVcpu) {
+      Warning("(%s) Offset %"FMT64"u (type %u, per-VCPU size %"FMT64"u)\n",
+              __FUNCTION__, pgOffset, type, region->pagesPerVcpu);
+      return INVALID_MPN;
+   }
+
+   return pages[pgOffset];
+}

vmmon-only/common/statVarsVmmon.c

@@ -0,0 +1,282 @@
+/*********************************************************
+ * Copyright (C) 2018-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * statVarsVmmon.c --
+ *
+ *     VMMon stat variables management.
+ */
+
+#ifdef __linux__
+#   include "driver-config.h"
+#   include <linux/string.h> /* memset() in the kernel. */
+#else
+#   include <string.h>
+#endif
+#include "vmware.h"
+#include "vm_assert.h"
+#include "hostif.h"
+#include "statVarsVmmon.h"
+#include "vmx86.h"
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * StatVarsVmmon_Init --
+ *
+ *      Initializes the stat vars component of the VM Driver.
+ *
+ * Results:
+ *      A pointer to the VMMon stat vars struct.
+ *
+ *----------------------------------------------------------------------
+ */
+
+StatVarsVmmon *
+StatVarsVmmon_Init(VMDriver *vm)
+{
+   StatVarsVmmon *statVars;
+
+   statVars = HostIF_AllocKernelMem(sizeof *statVars, FALSE);
+   if (statVars == NULL) {
+      Warning("StatVars failed to allocate handle.\n");
+      return NULL;
+   }
+   memset(statVars, 0, sizeof *statVars);
+   return statVars;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * StatVarsVmmon_Cleanup --
+ *
+ *      Cleans up the stat vars component of the VM Driver by freeing all
+ *      previously allocated VMMon stat vars memory.
+ *
+ * Results:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+void
+StatVarsVmmon_Cleanup(StatVarsVmmon *statVars)
+{
+   if (statVars == NULL) {
+      return;
+   }
+
+   if (statVars->pages != NULL) {
+      HostIF_FreeKernelMem(statVars->pages);
+   }
+   HostIF_FreeKernelMem(statVars);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * StatVarsVmmonValidateRegistration --
+ *
+ *      Validate the given VMMon stat vars registration block.
+ *
+ * Results:
+ *      TRUE if the block is valid and can be used to register stat vars
+ *      backing pages for a VCPU, FALSE otherwise.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static Bool
+StatVarsVmmonValidateRegistration(VMDriver *vm,
+                                   VMStatVarsRegistrationBlock *block)
+{
+   Vcpuid vcpu      = block->vcpu;
+   PageCnt numPages = block->numPages;
+   PageCnt pagesPerVcpu;
+   MPN *pages;
+
+   if (vm == NULL           ||
+       vm->statVars == NULL ||
+       numPages == 0        ||
+       vcpu >= vm->numVCPUs) {
+      return FALSE;
+   }
+
+   pagesPerVcpu = vm->statVars->pagesPerVcpu;
+   pages = vm->statVars->pages;
+   if (pagesPerVcpu == 0) {
+      ASSERT(pages == NULL);
+      return TRUE;
+   } else {
+      return pagesPerVcpu == numPages &&
+             pages[vcpu * pagesPerVcpu] == INVALID_MPN;
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * StatVarsVmmon_RegisterVCPU --
+ *
+ *      Register stat vars backing pages for the given VCPU with VMMon.
+ *      A successful return value indicates that stat vars backing pages
+ *      for the given VCPU are locked and tracked by the VM Driver.
+ *
+ * Results:
+ *      TRUE if the backing pages were successfully registered for the
+ *      given VCPU, FALSE otherwise.
+ *
+ *----------------------------------------------------------------------
+ */
+
+Bool
+StatVarsVmmon_RegisterVCPU(VMDriver *vm,
+                           VMStatVarsRegistrationBlock *block)
+{
+   Vcpuid vcpu = block->vcpu;
+   PageCnt pagesPerVcpu = block->numPages;
+   StatVarsVmmon *statVars = vm->statVars;
+   MPN *pages;
+   PageCnt page;
+
+   HostIF_VMLock(vm, 45);
+   if (!StatVarsVmmonValidateRegistration(vm, block)) {
+      HostIF_VMUnlock(vm, 45);
+      return FALSE;
+   }
+   if (statVars->pagesPerVcpu == 0) {
+      PageCnt totalPages = pagesPerVcpu * vm->numVCPUs;
+      statVars->pages = HostIF_AllocKernelMem(totalPages * sizeof *pages,
+                                              FALSE);
+      if (statVars->pages == NULL) {
+         HostIF_VMUnlock(vm, 45);
+         return FALSE;
+      }
+      for (page = 0; page < totalPages; page++) {
+         statVars->pages[page] = INVALID_MPN;
+      }
+      statVars->pagesPerVcpu = pagesPerVcpu;
+   }
+
+   /*
+    * Validate that all pages in this VCPU's region are still invalid. Before
+    * unlocking, temporarily set the first page to 0 to prevent a malicious user
+    * from triggering an ASSERT by firing parallel ioctls for the same VCPU.
+    */
+   pages = statVars->pages + vcpu * pagesPerVcpu;
+   for (page = 0; page < pagesPerVcpu; page++) {
+      ASSERT(pages[page] == INVALID_MPN);
+   }
+   ASSERT_ON_COMPILE(INVALID_MPN != 0);
+   pages[0] = 0;
+   HostIF_VMUnlock(vm, 45);
+
+   /*
+    * Verify that all stat vars backing pages are locked.  The pages are
+    * assumed to remain locked either until they are unlocked by the VMX
+    * following a NUMA migration, or by HostIF during VM Driver tear-down.
+    */
+   for (page = 0; page < pagesPerVcpu; page++) {
+      VPN vmxVPN = block->baseVpn + page;
+      VA64 uAddr = VPN_2_VA(vmxVPN);
+      PageCnt resetPage;
+      MPN *currPage = pages + page;
+      Bool verifiedLocked = FALSE;
+      int status = Vmx86_LockPage(vm, uAddr, FALSE, currPage);
+
+      /*
+       * Backing pages are expected to be locked by the VMX before being
+       * registered.
+       */
+      if (status == PAGE_LOCK_ALREADY_LOCKED) {
+         verifiedLocked = TRUE;
+         status = Vmx86_LookupUserMPN(vm, uAddr, currPage);
+         /*
+          * PR 2260615: Some platforms return PAGE_LOCK_SUCCESS for a successful
+          * user MPN lookup, others return PAGE_LOOKUP_SUCCESS, but they both
+          * happen to equal the same value.
+          */
+         ASSERT_ON_COMPILE(PAGE_LOOKUP_SUCCESS == PAGE_LOCK_SUCCESS);
+         if (status == PAGE_LOOKUP_SUCCESS) {
+            /*
+             * The backing page was verified as previously locked, and was
+             * successfully recorded.
+             */
+            continue;
+         }
+      }
+      /*
+       * The backing page was either not previously locked, or was not recorded.
+       * In either case, reset all previously recorded pages to INVALID_MPN.
+       */
+      if (!verifiedLocked && status == PAGE_LOCK_SUCCESS) {
+         status = Vmx86_UnlockPage(vm, uAddr);
+         ASSERT(status == PAGE_UNLOCK_SUCCESS);
+      }
+      for (resetPage = 0; resetPage <= page; resetPage++) {
+         pages[resetPage] = INVALID_MPN;
+      }
+      return FALSE;
+   }
+
+   return TRUE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * StatVarsVmmon_GetRegionMPN --
+ *
+ *      Obtain the backing MPN for stat vars for the given VCPU at
+ *      the given offset.
+ *
+ * Results:
+ *      The backing stat vars MPN for the given VCPU at the given offset
+ *      if it has been registered, or INVALID_MPN otherwise.
+ *
+ *----------------------------------------------------------------------
+ */
+
+MPN
+StatVarsVmmon_GetRegionMPN(struct VMDriver *vm, Vcpuid vcpu, PageCnt offset)
+{
+   MPN *pages;
+   MPN backingPage;
+   StatVarsVmmon *statVars = vm->statVars;
+
+   ASSERT(vcpu < vm->numVCPUs);
+
+   HostIF_VMLock(vm, 46);
+   if (offset >= statVars->pagesPerVcpu) {
+      HostIF_VMUnlock(vm, 46);
+      Warning("(%s) Requested MPN at invalid offset %"FMT64"u for VCPU %u\n",
+              __FUNCTION__, offset, vcpu);
+      return INVALID_MPN;
+   }
+
+   pages = statVars->pages + vcpu * statVars->pagesPerVcpu;
+   backingPage = pages[offset];
+   HostIF_VMUnlock(vm, 46);
+   return backingPage;
+}

vmmon-only/common/task.h

@@ -0,0 +1,42 @@
+/*********************************************************
+ * Copyright (C) 1998-2013,2015,2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef TASK_H
+#define TASK_H
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+#include "bootstrap_vmm.h"
+
+struct MonLoaderHeader;
+
+extern Bool Task_CreateCrossGDT(BSVMM_GDTInit *gdt);
+extern Bool Task_InitCrosspage(VMDriver *vm, LPN monStartLPN, LPN monEndLPN,
+                               PerVcpuPages *perVcpuPages);
+extern void Task_Switch(VMDriver *vm, Vcpuid vcpuid);
+extern Bool Task_Initialize(void);
+extern void Task_Terminate(void);
+extern MPN Task_GetHVRootPageForPCPU(uint32 pCPU);
+extern Descriptor *Task_GetTmpGDT(uint32 pCPU);
+extern void Task_SwitchPTPPageCleanup(VMDriver *vm);
+
+#endif
+
+
+

vmmon-only/common/vmx86.h

@@ -0,0 +1,234 @@
+/*********************************************************
+ * Copyright (C) 1998-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ *  vmx86.h - Platform independent data and interface for supporting
+ *            the vmx86 device driver.
+ */
+
+#ifndef VMX86_H
+#define VMX86_H
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#include "x86apic.h"
+#include "x86msr.h"
+#include "modulecall.h"
+#include "vcpuid.h"
+#include "iocontrols.h"
+#include "numa_defs.h"
+#include "rateconv.h"
+#include "vmmem_shared.h"
+#include "apic.h"
+#include "bootstrap_vmm.h"
+#include "sharedAreaVmmon.h"
+#include "statVarsVmmon.h"
+
+typedef struct TSCDelta {
+   Atomic_uint64 min;
+   Atomic_uint64 max;
+} TSCDelta;
+
+struct VmmBlobInfo;
+
+struct HostIFContigMemMap;
+
+extern struct HostIFContigMemMap *hvIOBitmap;
+
+/*
+ * VMDriver - the main data structure for the driver side of a
+ *            virtual machine.
+ */
+
+typedef struct VMDriver {
+   /* Unique (in the driver), strictly positive, VM ID used by userland. */
+   int                     userID;
+   Vcpuid                  numVCPUs;         /* Number of vcpus in VM. */
+   struct VMDriver        *nextDriver;       /* Next on list of all VMDrivers */
+   struct VMHost          *vmhost;           /* Host-specific fields. */
+   MPN                    *ptRootMpns;       /* numVCPUs-sized array. */
+   struct VmmBlobInfo     *blobInfo;         /* VMM bootstrap blob info. */
+   SharedAreaVmmon        *sharedArea;       /* VMMon shared area info. */
+   StatVarsVmmon          *statVars;         /* VMMon stat vars info. */
+   /* Pointers to the crossover pages shared with the monitor. */
+   struct VMCrossPage    **crosspage;        /* numVCPUs-sized array. */
+   struct MemTrack        *ptpTracker;       /* Tracks page table patch pages */
+   struct MemTrack        *vmmTracker;       /* Tracks allocated VMM pages */
+   VCPUSet                *crosscallWaitSet; /* numVCPUs-sized array. */
+   APICDescriptor          hostAPIC;
+   struct MemTrack        *memtracker;       /* Memory tracker pointer */
+   Bool                    checkFuncFailed;
+   struct PerfCounter     *perfCounter;
+   VMMemMgmtInfo           memInfo;
+   unsigned                fastClockRate;    /* Protected by FastClockLock. */
+   Atomic_uint64           ptscOffsetInfo;   /* Volatile per PR 699101#29. */
+   Atomic_uint64           ptscLatest;
+   int64                  *ptscOffsets;      /* numVCPUs-sized array. */
+   Atomic_uint32          *currentHostCpu;   /* numVCPUs-sized array. */
+   PageCnt                 numPTPPages;      /* Num PTP pages allocated. */
+   /*
+    * List of physically contiguous allocations associated with this VM.
+    * Access is protected by the VM lock.
+    */
+   struct HostIFContigMemMap     *contigMappings;
+
+} VMDriver;
+
+typedef struct MonLoaderArgs {
+   VMDriver *vm;
+   VMSharedRegion *shRegions;
+} MonLoaderArgs;
+
+typedef struct VmTimeStart {
+   uint64 count;
+   uint64 time;
+} VmTimeStart;
+
+typedef struct RefClockParams {
+   RateConv_Ratio ratio;
+   Atomic_uint64  add;
+} RefClockParams;
+
+typedef struct PseudoTSC {
+   RefClockParams refClockToPTSC;
+   int64          tscOffset;
+   uint64         hz;
+   volatile Bool  useRefClock;
+   Bool           neverSwitchToRefClock;
+   Bool           hwTSCsSynced;
+   volatile Bool  initialized;
+} PseudoTSC;
+
+extern PseudoTSC pseudoTSC;
+
+#define MAX_LOCKED_PAGES MAX_PPN
+
+extern void Vmx86_CacheNXState(void);
+extern VMDriver *Vmx86_CreateVM(VA64 bsBlob,
+                                uint32 bsBlobSize,
+                                uint32 numVCPUs);
+extern Bool Vmx86_ProcessBootstrap(VMDriver *vm,
+                                   VA64 bsBlobAddr,
+                                   uint32 numBytes,
+                                   uint32 headerOffset,
+                                   uint16 numVCPUs,
+                                   PerVcpuPages *perVcpuPages,
+                                   VMSharedRegion *shRegions);
+extern int Vmx86_LookupUserMPN(VMDriver *vm, VA64 uAddr, MPN *mpn);
+extern int Vmx86_ReleaseVM(VMDriver *vm);
+extern int Vmx86_LateInitVM(VMDriver *vm);
+extern int Vmx86_RunVM(VMDriver *vm, Vcpuid vcpuid);
+extern void Vmx86_YieldToSet(VMDriver *vm, Vcpuid currVcpu, const VCPUSet *req,
+                             uint32 usecs, Bool skew);
+extern void Vmx86_ReadTSCAndUptime(VmTimeStart *st);
+extern uint32 Vmx86_ComputekHz(uint64 cycles, uint64 uptime);
+extern uint32 Vmx86_GetkHzEstimate(VmTimeStart *st);
+extern int Vmx86_SetHostClockRate(VMDriver *vm, unsigned rate);
+extern int Vmx86_LockPage(VMDriver *vm,
+                          VA64 uAddr,
+                          Bool allowMultipleMPNsPerVA,
+                          MPN *mpn);
+extern int Vmx86_UnlockPage(VMDriver *vm, VA64 uAddr);
+extern int Vmx86_UnlockPageByMPN(VMDriver *vm, MPN mpn, VA64 uAddr);
+extern MPN Vmx86_GetRecycledPage(VMDriver *vm);
+extern int Vmx86_ReleaseAnonPage(VMDriver *vm, MPN mpn);
+extern int64 Vmx86_AllocLockedPages(VMDriver *vm, VA64 addr,
+                                    PageCnt numPages, Bool kernelMPNBuffer,
+                                    Bool ignoreLimits);
+extern int Vmx86_FreeLockedPages(VMDriver *vm, MPN *mpns, PageCnt numPages);
+extern MPN Vmx86_GetNextAnonPage(VMDriver *vm, MPN mpn);
+extern MPN Vmx86_GetNumAnonPages(VMDriver *vm);
+extern MPN Vmx86_AllocLowPage(VMDriver *vm, Bool ignoreLimits);
+extern void *Vmx86_Calloc(size_t numElements,
+                          size_t elementSize,
+                          Bool nonPageable);
+extern void  Vmx86_Free(void *ptr);
+
+extern int32 Vmx86_GetNumVMs(void);
+extern Bool Vmx86_SetConfiguredLockedPagesLimit(PageCnt limit);
+extern void Vmx86_SetDynamicLockedPagesLimit(PageCnt limit);
+extern Bool Vmx86_GetMemInfo(VMDriver *curVM,
+                             Bool curVMOnly,
+                             VMMemInfoArgs *args,
+                             int outArgsLength);
+extern void Vmx86_Admit(VMDriver *curVM, VMMemInfoArgs *args);
+extern Bool Vmx86_Readmit(VMDriver *curVM, OvhdMem_Deltas *delta);
+extern void Vmx86_UpdateMemInfo(VMDriver *curVM,
+                                const VMMemMgmtInfoPatch *patch);
+extern void Vmx86_Add2MonPageTable(VMDriver *vm, VPN vpn, MPN mpn,
+                                   Bool readOnly);
+extern Bool Vmx86_GetAllMSRs(MSRQuery *query);
+extern void Vmx86_FlushVMCSAllCPUs(MA vmcs);
+extern void Vmx86_MonTimerIPI(void);
+extern void Vmx86_InitIDList(void);
+extern Bool Vmx86_CreateHVIOBitmap(void);
+extern void Vmx86_CleanupHVIOBitmap(void);
+extern Bool Vmx86_GetPageRoot(VMDriver *vm, Vcpuid vcpuid, MPN *mpn);
+extern void Vmx86_Open(void);
+extern void Vmx86_Close(void);
+
+static INLINE Bool
+Vmx86_HwTSCsSynced(void)
+{
+   return pseudoTSC.hwTSCsSynced;
+}
+
+static INLINE Bool
+Vmx86_PseudoTSCUsesRefClock(void)
+{
+   return pseudoTSC.useRefClock;
+}
+
+static INLINE Bool
+Vmx86_SetPseudoTSCUseRefClock(void)
+{
+   if (!pseudoTSC.useRefClock && !pseudoTSC.neverSwitchToRefClock) {
+      pseudoTSC.useRefClock = TRUE;
+      return TRUE;
+   }
+   return FALSE;
+}
+
+static INLINE uint64
+Vmx86_GetPseudoTSCHz(void)
+{
+   return pseudoTSC.hz;
+}
+
+static INLINE uint64
+Vmx86_GetPseudoTSCOffset(void)
+{
+   return pseudoTSC.tscOffset;
+}
+
+extern void Vmx86_InitPseudoTSC(PTSCInitParams *params);
+extern Bool Vmx86_CheckPseudoTSC(uint64 *lastTSC, uint64 *lastRC);
+extern uint64 Vmx86_GetPseudoTSC(void);
+
+extern uint64 Vmx86_GetUnavailablePerfCtrs(void);
+
+extern Bool Vmx86_GetMonitorContext(VMDriver *vm, Vcpuid vcpuid,
+                                    Context64 *context);
+extern void Vmx86_CleanupVMMPages(VMDriver *vm);
+
+extern VPN Vmx86_MapPage(MPN mpn);
+extern void Vmx86_UnmapPage(VPN vpn);
+
+#endif

vmmon-only/include/address_defs.h

@@ -0,0 +1,105 @@
+/*********************************************************
+ * Copyright (C) 2014 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * address_defs.h --
+ *
+ *	Macros for virtual/physical/machine address/page conversions, page types.
+ */
+
+#ifndef _ADDRESS_DEFS_H_
+#define _ADDRESS_DEFS_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+#include "vm_basic_defs.h" // For PAGE_SHIFT
+
+#if defined __cplusplus
+extern "C" {
+#endif
+
+
+/*
+ * Virtual, physical, machine address and page conversion macros
+ */
+
+#define VA_2_VPN(_va)  ((_va) >> PAGE_SHIFT)
+#define PTR_2_VPN(_ptr) VA_2_VPN((VA)(_ptr))
+#define VPN_2_VA(_vpn) ((_vpn) << PAGE_SHIFT)
+#define VPN_2_PTR(_vpn) ((void *)VPN_2_VA(_vpn))
+
+/*
+ * Notice that we don't cast PA_2_PPN's argument to an unsigned type, because
+ * we would lose compile-time checks for pointer operands and byte-sized
+ * operands. If you use a signed quantity for _pa, ones may be shifted into the
+ * high bits of your ppn.
+ */
+
+#define PA_2_PPN(_pa)     ((_pa) >> PAGE_SHIFT)
+#define PPN_2_PA(_ppn)    ((PA)(_ppn) << PAGE_SHIFT)
+
+static INLINE MA    MPN_2_MA(MPN mpn)     { return  (MA)mpn << PAGE_SHIFT;  }
+static INLINE MPN   MA_2_MPN(MA ma)       { return (MPN)(ma >> PAGE_SHIFT); }
+
+static INLINE IOA   IOPN_2_IOA(IOPN iopn) { return (IOA)(iopn << PAGE_SHIFT); }
+static INLINE IOPN  IOA_2_IOPN(IOA ioa)   { return (IOPN)(ioa >> PAGE_SHIFT); }
+
+typedef char PageArray[PAGE_SIZE];
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * IsGoodMPN --
+ *
+ *      Is the given MPN valid?
+ *
+ * Results:
+ *      Return TRUE if "mpn" looks plausible. We could make this stricter on
+ *      a per-architecture basis.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+IsGoodMPN(MPN mpn)
+{
+   return mpn <= MAX_MPN;
+}
+
+static INLINE Bool
+IsGoodMPNOrMemref(MPN mpn)
+{
+   return IsGoodMPN(mpn) || mpn == MEMREF_MPN;
+}
+
+
+#if defined __cplusplus
+} // extern "C"
+#endif
+
+#endif // _ADDRESS_DEFS_H_

vmmon-only/include/bootstrap_vmm.h

@@ -0,0 +1,65 @@
+/*********************************************************
+ * Copyright (C) 2015,2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * bootstrap_vmm.h --
+ *
+ *    Public VMM bootstrap declarations.
+ */
+
+#ifndef _BOOTSTRAP_VMM_H
+#define _BOOTSTRAP_VMM_H
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_USERLEVEL
+
+#include "includeCheck.h"
+#include "vm_basic_types.h"
+#include "monAddrLayout.h"
+
+#define BOOTSTRAP_MAGIC 42
+#define BOOTSTRAP_MAX_GDT_DESCS 2
+
+typedef struct BSVMM_GDTInitEntry {
+   uint16 index;
+   LA32   base;
+   VA32   limit;
+   uint32 type;
+   uint32 S;
+   uint32 DPL;
+   uint32 present;
+   uint32 longmode;
+   uint32 DB;
+   uint32 gran;
+} BSVMM_GDTInitEntry;
+
+typedef struct BSVMM_GDTInit {
+   BSVMM_GDTInitEntry entries[BOOTSTRAP_MAX_GDT_DESCS];
+} BSVMM_GDTInit;
+
+typedef struct BSVMM_HostParams {
+   uint32           magic;
+   VMM64_AddrLayout addrLayout;
+   BSVMM_GDTInit    gdtInit;
+} BSVMM_HostParams;
+
+BSVMM_HostParams* BSVMM_Validate(void *buf, uint32 nbytes);
+
+#endif

vmmon-only/include/circList.h

@@ -0,0 +1,507 @@
+/*********************************************************
+ * Copyright (C) 1998-2017,2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ *   circList.h --
+ *
+ * macros, prototypes and struct definitions for double-linked
+ * circular lists.
+ */
+
+#ifndef _CIRCLIST_H_
+#define _CIRCLIST_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#include "includeCheck.h"
+
+#include "vmware.h"
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+typedef struct ListItem {
+   struct ListItem *prev;
+   struct ListItem *next;
+} ListItem;
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_IsEmpty --
+ *
+ *      A NULL list is an empty list.
+ *
+ * Result:
+ *      TRUE if list is empty, FALSE otherwise.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+CircList_IsEmpty(const ListItem *item)  // IN
+{
+   return item == NULL;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_InitItem --
+ *
+ *      Initialize item as a single-element circular list.
+ *
+ * Result:
+ *      None.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+CircList_InitItem(ListItem *item)  // OUT
+{
+   item->prev = item->next = item;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_First --
+ *
+ *      Return first item in the list.
+ *
+ * Result:
+ *      First item.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE ListItem *
+CircList_First(ListItem *item)  // IN
+{
+   return item;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_Last --
+ *
+ *      Return last item in the list.
+ *
+ * Result:
+ *      Last item.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE ListItem *
+CircList_Last(ListItem *item)
+{
+   return item->prev;
+}
+
+
+/*
+ * CIRC_LIST_CONTAINER - get the struct for this entry (like list_entry)
+ * @ptr: the &struct ListItem pointer.
+ * @type:   the type of the struct this is embedded in.
+ * @member: the name of the list struct within the struct.
+ */
+#define CIRC_LIST_CONTAINER(ptr, type, member) \
+   VMW_CONTAINER_OF(ptr, type, member)
+/* 
+ * Historical name, left here to reduce churn.
+ * TODO: remove, all LIST_CONTAINER uses should be
+ * VMW_CONTAINER_OF and stop depending on circList.h
+ * to provide the definition.
+ */
+#define LIST_CONTAINER(ptr, type, member) VMW_CONTAINER_OF(ptr, type, member)
+
+/*
+ * LIST_SCAN_FROM scans the list from "from" up until "until".
+ * The loop variable p should not be destroyed in the process.
+ * "from" is an element in the list where to start scanning.
+ * "until" is the element where search should stop.
+ * member is the field to use for the search - either "next" or "prev".
+ */
+#define CIRC_LIST_SCAN_FROM(p, from, until, member)   \
+   for (p = (from); (p) != NULL;   \
+      (p) = (((p)->member == (until)) ? NULL : (p)->member))
+
+/* scan the entire list (non-destructively) */
+#define CIRC_LIST_SCAN(p, l)   \
+   CIRC_LIST_SCAN_FROM(p, CircList_First(l), CircList_First(l), next)
+
+
+/* scan the entire list where loop element may be destroyed */
+#define CIRC_LIST_SCAN_SAFE(p, pn, l)   \
+   if (!CircList_IsEmpty(l))  \
+      for (p = (l), (pn) = CircList_Next(p, l); (p) != NULL;   \
+           (p) = (pn), (pn) = CircList_Next(p, l))
+
+/* scan the entire list backwards where loop element may be destroyed */
+#define CIRC_LIST_SCAN_BACK_SAFE(p, pn, l)   \
+   if (!CircList_IsEmpty(l))  \
+      for (p = CircList_Last(l), (pn) = CircList_Prev(p, l); (p) != NULL;   \
+           (p) = (pn), (pn) = CircList_Prev(p, l))
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_Next --
+ *
+ *      Returns the next member of a doubly linked list, or NULL if last.
+ *      Assumes: p is member of the list headed by head.
+ *
+ * Result:
+ *      If head or p is NULL, return NULL. Otherwise,
+ *      next list member (or null if last).
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE ListItem *
+CircList_Next(ListItem *p,        // IN
+              ListItem *head)     // IN
+{
+   if (head == NULL || p == NULL) {
+      return NULL;
+   }
+   /* both p and head are non-null */
+   p = p->next;
+   return p == head ? NULL : p;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_Prev --
+ *
+ *      Returns the prev member of a doubly linked list, or NULL if first.
+ *      Assumes: p is member of the list headed by head.
+ *
+ * Result:
+ *      If head or prev is NULL, return NULL. Otherwise,
+ *      prev list member (or null if first).
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE ListItem *
+CircList_Prev(ListItem *p,        // IN
+              ListItem *head)     // IN
+{
+   if (head == NULL || p == NULL) {
+      return NULL;
+   }
+   /* both p and head are non-null */
+   return p == head ? NULL : p->prev;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_DeleteItem --
+ *
+ *      Deletes a member of a doubly linked list, possibly modifies the
+ *      list header itself.
+ *      Assumes neither p nor headp is null and p is a member of *headp.
+ *
+ * Result:
+ *      None
+ *
+ * Side effects:
+ *      Modifies *headp.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+CircList_DeleteItem(ListItem *p,         // IN
+                    ListItem **headp)    // IN/OUT
+{
+   ListItem *next;
+
+   ASSERT(p != NULL);
+   ASSERT(headp != NULL);
+
+   next = p->next;
+   if (p == next) {
+      *headp = NULL;
+   } else {
+      next->prev = p->prev;
+      p->prev->next = next;
+      if (*headp == p) {
+         *headp = next;
+      }
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_Queue --
+ *
+ *      Adds a new member to the back of a doubly linked list (queue)
+ *      Assumes neither p nor headp is null and p is not a member of *headp.
+ *
+ * Result:
+ *      None
+ *
+ * Side effects:
+ *      Modifies *headp.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+CircList_Queue(ListItem *p,              // IN
+               ListItem **headp)         // IN/OUT
+{
+   ListItem *head;
+
+   head = *headp;
+   if (CircList_IsEmpty(head)) {
+      CircList_InitItem(p);
+      *headp = p;
+   } else {
+      p->prev = head->prev;
+      p->next = head;
+      p->prev->next = p;
+      head->prev = p;
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_Push --
+ *
+ *      Adds a new member to the front of a doubly linked list (stack)
+ *      Assumes neither p nor headp is null and p is not a member of *headp.
+ *
+ * Result:
+ *      None
+ *
+ * Side effects:
+ *      Modifies *headp.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+CircList_Push(ListItem *p,               // IN
+              ListItem **headp)          // IN/OUT
+{
+   CircList_Queue(p, headp);
+   *headp = p;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_InsertAfter --
+ *
+ *      Adds a new member to the list after the provided item.  Assumes p
+ *      is not a member of a list already.
+ *
+ * Result:
+ *      None.
+ *
+ * Side effects:
+ *      None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+CircList_InsertAfter(ListItem *p,      // IN:
+                     ListItem *after)  // IN:
+{
+   p->prev = after;
+   p->next = after->next;
+   p->next->prev = p;
+   after->next = p;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_Splice --
+ *
+ *      Make a single list {l1 l2} from {l1} and {l2} and return it.
+ *      It is okay for one or both lists to be NULL.
+ *      No checking is done. It is assumed that l1 and l2 are two
+ *      distinct lists.
+ *
+ * Result:
+ *      A list { l1 l2 }.
+ *
+ * Side effects:
+ *      Modifies l1 and l2 list pointers.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE ListItem *
+CircList_Splice(ListItem *l1,      // IN
+                ListItem *l2)      // IN
+{
+   ListItem *l1Last, *l2Last;
+
+   if (CircList_IsEmpty(l1)) {
+      return l2;
+   }
+
+   if (CircList_IsEmpty(l2)) {
+      return l1;
+   }
+
+   l1Last = l1->prev;   /* last elem of l1 */
+   l2Last = l2->prev;   /* last elem of l2 */
+
+   /*
+    *    l1 -> ... -> l1Last    l2 -> ... l2Last
+    */
+   l1Last->next = l2;
+   l2->prev = l1Last;
+
+   l1->prev = l2Last;
+   l2Last->next = l1;
+
+   return l1;
+}
+
+
+#if 0  /* Presently unused, enable if a use is found */
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_Split --
+ *
+ *      Make a list l = {l1 l2} into two separate lists {l1} and {l2}, where:
+ *      l = { ... x -> p -> ... } split into:
+ *      l1 = { ... -> x }
+ *      l2 = { p -> ... }
+ *      Assumes neither p nor l is null and p is a member of l.
+ *      If p is the first element of l, then l1 will be NULL.
+ *
+ * Result:
+ *      None.
+ *
+ * Side effects:
+ *      Sets *l1p and *l2p to the resulting two lists.
+ *      Modifies l's pointers.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+CircList_Split(ListItem *p,         // IN
+               ListItem *l,         // IN
+               ListItem **l1p,      // OUT
+               ListItem **l2p)      // OUT
+{
+   ListItem *last;
+
+   if (p == CircList_First(l)) {   /* first element */
+      *l1p = NULL;
+      *l2p = l;
+      return;
+   }
+
+   last = l->prev;
+
+   *l1p = l;
+   p->prev->next = l;
+   l->prev = p->prev;
+
+   *l2p = p;
+   p->prev = last;
+   last->next = p;
+}
+#endif
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * CircList_Size --
+ *
+ *	Return the number of items in the list.
+ *
+ * Result:
+ *	The number of items in the list.
+ *
+ * Side effects:
+ *	None.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE int
+CircList_Size(ListItem *head)     // IN
+{
+   ListItem *li;
+   int ret = 0;
+
+   CIRC_LIST_SCAN(li, head) {
+      ret++;
+   }
+   return ret;
+}
+
+#if defined(__cplusplus)
+}  // extern "C"
+#endif
+
+#endif /* _CIRCLIST_H_ */

vmmon-only/include/community_source.h

@@ -0,0 +1,67 @@
+/*********************************************************
+ * Copyright (C) 2009-2016 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * community_source.h --
+ *
+ *    Macros for excluding source code from community.
+ */
+
+#ifndef _COMMUNITY_SOURCE_H_
+#define _COMMUNITY_SOURCE_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMKDRIVERS
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+/* 
+ * Convenience macro for COMMUNITY_SOURCE
+ */
+#undef EXCLUDE_COMMUNITY_SOURCE
+#ifdef COMMUNITY_SOURCE
+   #define EXCLUDE_COMMUNITY_SOURCE(x) 
+#else
+   #define EXCLUDE_COMMUNITY_SOURCE(x) x
+#endif
+
+#undef COMMUNITY_SOURCE_AMD_SECRET
+#if !defined(COMMUNITY_SOURCE) || defined(AMD_SOURCE)
+/*
+ * It's ok to include AMD_SECRET source code for non-Community Source,
+ * or for drops directed at AMD.
+ */
+   #define COMMUNITY_SOURCE_AMD_SECRET
+#endif
+
+#undef COMMUNITY_SOURCE_INTEL_SECRET
+#if !defined(COMMUNITY_SOURCE) || defined(INTEL_SOURCE)
+/*
+ * It's ok to include INTEL_SECRET source code for non-Community Source,
+ * or for drops directed at Intel.
+ */
+   #define COMMUNITY_SOURCE_INTEL_SECRET
+#endif
+
+#endif

vmmon-only/include/compat_autoconf.h

@@ -0,0 +1,41 @@
+/*********************************************************
+ * Copyright (C) 2009 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef __COMPAT_AUTOCONF_H__
+#   define __COMPAT_AUTOCONF_H__
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMKDRIVERS
+#include "includeCheck.h"
+
+
+#ifndef LINUX_VERSION_CODE
+#   error "Include compat_version.h before compat_autoconf.h"
+#endif
+
+/* autoconf.h moved from linux/autoconf.h to generated/autoconf.h in 2.6.33-rc1. */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 33)
+#   include <linux/autoconf.h>
+#else
+#   include <generated/autoconf.h>
+#endif
+
+#endif /* __COMPAT_AUTOCONF_H__ */

vmmon-only/include/compat_kernel.h

@@ -0,0 +1,42 @@
+/*********************************************************
+ * Copyright (C) 2004 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef __COMPAT_KERNEL_H__
+#   define __COMPAT_KERNEL_H__
+
+#include <asm/unistd.h>
+#include <linux/kernel.h>
+
+/*
+ * container_of was introduced in 2.5.28 but it's easier to check like this.
+ */
+#ifndef container_of
+#define container_of(ptr, type, member) ({			\
+        const typeof( ((type *)0)->member ) *__mptr = (ptr);	\
+        (type *)( (char *)__mptr - offsetof(type,member) );})
+#endif
+
+/*
+ * vsnprintf became available in 2.4.10. For older kernels, just fall back on
+ * vsprintf.
+ */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 4, 10)
+#define vsnprintf(str, size, fmt, args) vsprintf(str, fmt, args)
+#endif
+
+#endif /* __COMPAT_KERNEL_H__ */

vmmon-only/include/compat_module.h

@@ -0,0 +1,83 @@
+/*********************************************************
+ * Copyright (C) 2007 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * compat_module.h --
+ */
+
+#ifndef __COMPAT_MODULE_H__
+#   define __COMPAT_MODULE_H__
+
+
+#include <linux/module.h>
+
+
+/*
+ * Modules wishing to use the GPL license are required to include a
+ * MODULE_LICENSE definition in their module source as of 2.4.10.
+ */
+#ifndef MODULE_LICENSE
+#define MODULE_LICENSE(license)
+#endif
+
+/*
+ * To make use of our own home-brewed MODULE_INFO, we need macros to
+ * concatenate two expressions to "__mod_", and and to convert an
+ * expression into a string. I'm sure we've got these in our codebase,
+ * but I'd rather not introduce such a dependency in a compat header.
+ */
+#ifndef __module_cat
+#define __module_cat_1(a, b) __mod_ ## a ## b
+#define __module_cat(a, b) __module_cat_1(a, b)
+#endif
+
+#ifndef __stringify
+#define __stringify_1(x) #x
+#define __stringify(x) __stringify_1(x)
+#endif
+
+/*
+ * MODULE_INFO was born in 2.5.69.
+ */
+#ifndef MODULE_INFO
+#define MODULE_INFO(tag, info)                                                \
+static const char __module_cat(tag, __LINE__)[]                               \
+  __attribute__((section(".modinfo"), unused)) = __stringify(tag) "=" info
+#endif
+
+/*
+ * MODULE_VERSION was born in 2.6.4. The earlier form appends a long "\0xxx"
+ * string to the module's version, but that was removed in 2.6.10, so we'll
+ * ignore it in our wrapper.
+ */
+#ifndef MODULE_VERSION
+#define MODULE_VERSION(_version) MODULE_INFO(version, _version)
+#endif
+
+/*
+ * Linux kernel < 2.6.31 takes 'int' for 'bool' module parameters.
+ * Linux kernel >= 3.3.0 takes 'bool' for 'bool' module parameters.
+ * Kernels between the two take either.  So flip switch at 3.0.0.
+ */
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 0, 0)
+   typedef bool compat_mod_param_bool;
+#else
+   typedef int compat_mod_param_bool;
+#endif
+
+#endif /* __COMPAT_MODULE_H__ */

vmmon-only/include/compat_page.h

@@ -0,0 +1,75 @@
+/*********************************************************
+ * Copyright (C) 2002 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef __COMPAT_PAGE_H__
+#   define __COMPAT_PAGE_H__
+
+
+#include <linux/mm.h>
+#include <asm/page.h>
+
+
+/* The pfn_to_page() API appeared in 2.5.14 and changed to function during 2.6.x */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 0) && !defined(pfn_to_page)
+#   define pfn_to_page(_pfn) (mem_map + (_pfn))
+#   define page_to_pfn(_page) ((_page) - mem_map)
+#endif
+
+
+/* The virt_to_page() API appeared in 2.4.0 --hpreg */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 4, 0) && !defined(virt_to_page)
+#   define virt_to_page(_kvAddr) pfn_to_page(MAP_NR(_kvAddr))
+#endif
+
+
+/*
+ * The get_order() API appeared at some point in 2.3.x, and was then backported
+ * in 2.2.17-21mdk and in the stock 2.2.18. Because we can only detect its
+ * definition through makefile tricks, we provide our own for now --hpreg
+ */
+static inline int
+compat_get_order(unsigned long size) // IN
+{
+   int order;
+
+   size = (size - 1) >> (PAGE_SHIFT - 1);
+   order = -1;
+   do {
+      size >>= 1;
+      order++;
+   } while (size);
+
+   return order;
+}
+
+/* 
+ * BUG() was added to <asm/page.h> in 2.2.18, and was moved to <asm/bug.h>
+ * in 2.5.58.
+ * 
+ * XXX: Technically, this belongs in some sort of "compat_asm_page.h" file, but
+ * since our compatibility wrappers don't distinguish between <asm/xxx.h> and
+ * <linux/xxx.h>, putting it here is reasonable.
+ */
+#ifndef BUG
+#define BUG() do {                                                            \
+   printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__);                      \
+  __asm__ __volatile__(".byte 0x0f,0x0b");                                    \
+} while (0)
+#endif
+
+#endif /* __COMPAT_PAGE_H__ */

vmmon-only/include/compat_pgtable.h

@@ -0,0 +1,82 @@
+/*********************************************************
+ * Copyright (C) 2002-2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef __COMPAT_PGTABLE_H__
+#   define __COMPAT_PGTABLE_H__
+
+
+#if defined(CONFIG_PARAVIRT) && defined(CONFIG_HIGHPTE)
+#   if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 21)
+#      include <asm/paravirt.h>
+#      undef paravirt_map_pt_hook
+#      define paravirt_map_pt_hook(type, va, pfn) do {} while (0)
+#   endif
+#endif
+#include <asm/pgtable.h>
+
+
+/*
+ * p4d level appeared in 4.12.
+ */
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 12, 0)
+#   define compat_p4d_offset(pgd, address)  p4d_offset(pgd, address)
+#   define compat_p4d_present(p4d)          p4d_present(p4d)
+#   define compat_p4d_large(p4d)            p4d_large(p4d)
+#   define compat_p4d_pfn(p4d)              p4d_pfn(p4d)
+#   define COMPAT_P4D_MASK                  P4D_MASK
+typedef p4d_t compat_p4d_t;
+#else
+#   define compat_p4d_offset(pgd, address)  (pgd)
+#   define compat_p4d_present(p4d)          (1)
+#   define compat_p4d_large(p4d)            (0)
+#   define compat_p4d_pfn(p4d)              INVALID_MPN  /* Not used */
+#   define COMPAT_P4D_MASK                  0            /* Not used */
+typedef pgd_t compat_p4d_t;
+#endif
+/* pud_pfn did not exist before 3.8. */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 8, 0)
+#   define pud_pfn(pud)  INVALID_MPN
+#endif
+
+
+/*
+ * Define VM_PAGE_KERNEL_EXEC for vmapping executable pages.
+ *
+ * On ia32 PAGE_KERNEL_EXEC was introduced in 2.6.8.1.  Unfortunately it accesses
+ * __PAGE_KERNEL_EXEC which is not exported for modules.  So we use
+ * __PAGE_KERNEL and just cut _PAGE_NX bit from it.
+ *
+ * For ia32 kernels before 2.6.8.1 we use PAGE_KERNEL directly, these kernels
+ * do not have noexec support.
+ *
+ * On x86-64 situation is a bit better: they always supported noexec, but
+ * before 2.6.8.1 flag was named PAGE_KERNEL_EXECUTABLE, and it was renamed
+ * to PAGE_KERNEL_EXEC when ia32 got noexec too (see above).
+ */
+#ifdef CONFIG_X86
+#ifdef _PAGE_NX
+#define VM_PAGE_KERNEL_EXEC __pgprot(__PAGE_KERNEL & ~_PAGE_NX)
+#else
+#define VM_PAGE_KERNEL_EXEC PAGE_KERNEL
+#endif
+#else
+#define VM_PAGE_KERNEL_EXEC PAGE_KERNEL_EXEC
+#endif
+
+
+#endif /* __COMPAT_PGTABLE_H__ */

vmmon-only/include/compat_poll.h

@@ -0,0 +1,30 @@
+#ifndef __COMPAT_POLL_H__
+#define __COMPAT_POLL_H__
+
+#include <linux/poll.h>
+
+#if (LINUX_VERSION_CODE < KERNEL_VERSION(4, 18, 0))
+
+#ifndef __poll_t
+typedef unsigned int __poll_t;
+#endif
+
+static inline __poll_t compat_vfs_poll(struct file *file,
+				       struct poll_table_struct *pt)
+{
+	if (unlikely(!file->f_op->poll))
+		return DEFAULT_POLLMASK;
+	return file->f_op->poll(file, pt);
+}
+
+#else
+
+static inline __poll_t compat_vfs_poll(struct file *file,
+				       struct poll_table_struct *pt)
+{
+	return vfs_poll(file, pt);
+}
+
+#endif
+
+#endif /* __COMPAT_POLL_H__ */

vmmon-only/include/compat_sched.h

@@ -0,0 +1,293 @@
+/*********************************************************
+ * Copyright (C) 2002 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef __COMPAT_SCHED_H__
+#   define __COMPAT_SCHED_H__
+
+
+#include <linux/sched.h>
+
+/* CLONE_KERNEL available in 2.5.35 and higher. */
+#ifndef CLONE_KERNEL
+#define CLONE_KERNEL CLONE_FILES | CLONE_FS | CLONE_SIGHAND
+#endif
+
+/* TASK_COMM_LEN become available in 2.6.11. */
+#ifndef TASK_COMM_LEN
+#define TASK_COMM_LEN 16
+#endif
+
+/* The capable() API appeared in 2.1.92 --hpreg */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 1, 92)
+#   define capable(_capability) suser()
+#endif
+
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 2, 0)
+#   define need_resched() need_resched
+#elif LINUX_VERSION_CODE < KERNEL_VERSION(2, 5, 3)
+#   define need_resched() (current->need_resched)
+#endif
+
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 5, 3)
+#   define cond_resched() (need_resched() ? schedule() : (void) 0)
+#endif
+
+/* Oh well.  We need yield...  Happy us! */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 4, 20)
+#   ifdef __x86_64__
+#      define compat_yield() there_is_nothing_like_yield()
+#   else
+#      include <linux/unistd.h>
+#      include <linux/kernel.h>
+
+/*
+ * Used by _syscallX macros. Note that this is global variable, so
+ * do not rely on its contents too much. As exit() is only function
+ * we use, and we never check return value from exit(), we have
+ * no problem...
+ */
+extern int errno;
+
+/*
+ * compat_exit() provides an access to the exit() function. It must 
+ * be named compat_exit(), as exit() (with different signature) is 
+ * provided by x86-64, arm and other (but not by i386).
+ */
+#      define __NR_compat_yield __NR_sched_yield
+static inline _syscall0(int, compat_yield);
+#   endif
+#else
+#   define compat_yield() yield()
+#endif
+
+
+/*
+ * Since 2.5.34 there are two methods to enumerate tasks:
+ * for_each_process(p) { ... } which enumerates only tasks and
+ * do_each_thread(g,t) { ... } while_each_thread(g,t) which enumerates
+ *     also threads even if they share same pid.
+ */
+#ifndef for_each_process
+#   define for_each_process(p) for_each_task(p)
+#endif
+
+#ifndef do_each_thread
+#   define do_each_thread(g, t) for_each_task(g) { t = g; do
+#   define while_each_thread(g, t) while (0) }
+#endif
+
+
+/*
+ * Lock for signal mask is moving target...
+ */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 5, 40) && defined(CLONE_PID)
+/* 2.4.x without NPTL patches or early 2.5.x */
+#define compat_sigmask_lock sigmask_lock
+#define compat_dequeue_signal_current(siginfo_ptr) \
+   dequeue_signal(&current->blocked, (siginfo_ptr))
+#elif LINUX_VERSION_CODE < KERNEL_VERSION(2, 5, 60) && !defined(INIT_SIGHAND)
+/* RedHat's 2.4.x with first version of NPTL support, or 2.5.40 to 2.5.59 */
+#define compat_sigmask_lock sig->siglock
+#define compat_dequeue_signal_current(siginfo_ptr) \
+   dequeue_signal(&current->blocked, (siginfo_ptr))
+#else
+/* RedHat's 2.4.x with second version of NPTL support, or 2.5.60+. */
+#define compat_sigmask_lock sighand->siglock
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 5, 0)
+#define compat_dequeue_signal_current(siginfo_ptr) \
+   dequeue_signal(&current->blocked, (siginfo_ptr))
+#else
+#define compat_dequeue_signal_current(siginfo_ptr) \
+   dequeue_signal(current, &current->blocked, (siginfo_ptr))
+#endif
+#endif
+
+/*
+ * recalc_sigpending() had task argument in the past
+ */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 5, 29) && defined(CLONE_PID)
+/* 2.4.x without NPTL patches or early 2.5.x */
+#define compat_recalc_sigpending() recalc_sigpending(current)
+#else
+/* RedHat's 2.4.x with NPTL support, or 2.5.29+ */
+#define compat_recalc_sigpending() recalc_sigpending()
+#endif
+
+
+/*
+ * reparent_to_init() was introduced in 2.4.8.  In 2.5.38 (or possibly
+ * earlier, but later than 2.5.31) a call to it was added into
+ * daemonize(), so compat_daemonize no longer needs to call it.
+ *
+ * In 2.4.x kernels reparent_to_init() forgets to do correct refcounting
+ * on current->user. It is better to count one too many than one too few...
+ */
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 4, 8) && LINUX_VERSION_CODE < KERNEL_VERSION(2, 5, 38)
+#define compat_reparent_to_init() do { \
+					reparent_to_init(); \
+					atomic_inc(&current->user->__count); \
+				  } while (0)
+#else
+#define compat_reparent_to_init() do {} while (0)
+#endif
+
+
+/*
+ * daemonize appeared in 2.2.18. Except 2.2.17-4-RH7.0, which has it too.
+ * Fortunately 2.2.17-4-RH7.0 uses versioned symbols, so we can check
+ * its existence with defined().
+ */
+#if (LINUX_VERSION_CODE < KERNEL_VERSION(2, 2, 18)) && !defined(daemonize)
+static inline void daemonize(void) {
+   struct fs_struct *fs;
+
+   exit_mm(current);
+   current->session = 1;
+   current->pgrp = 1;
+   exit_fs(current);
+   fs = init_task.fs;
+   current->fs = fs;
+   atomic_inc(&fs->count);
+}
+#endif
+
+
+/*
+ * flush_signals acquires sighand->siglock since 2.5.61... Verify RH's kernels!
+ */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 5, 61)
+#define compat_flush_signals(task) do { \
+				      spin_lock_irq(&task->compat_sigmask_lock); \
+				      flush_signals(task); \
+				      spin_unlock_irq(&task->compat_sigmask_lock); \
+				   } while (0)
+#else
+#define compat_flush_signals(task) flush_signals(task)
+#endif
+
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 5, 61)
+#define compat_allow_signal(signr) do { \
+                                      spin_lock_irq(&current->compat_sigmask_lock); \
+                                      sigdelset(&current->blocked, signr); \
+                                      compat_recalc_sigpending(); \
+                                      spin_unlock_irq(&current->compat_sigmask_lock); \
+                                   } while (0)
+#else
+#define compat_allow_signal(signr) allow_signal(signr)
+#endif
+
+/*
+ * daemonize can set process name since 2.5.61. Prior to 2.5.61, daemonize
+ * didn't block signals on our behalf.
+ */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 5, 61)
+#define compat_daemonize(x...)                                                \
+({                                                                            \
+   /* Beware! No snprintf here, so verify arguments! */                       \
+   sprintf(current->comm, x);                                                 \
+                                                                              \
+   /* Block all signals. */                                                   \
+   spin_lock_irq(&current->compat_sigmask_lock);                              \
+   sigfillset(&current->blocked);                                             \
+   compat_recalc_sigpending();                                                \
+   spin_unlock_irq(&current->compat_sigmask_lock);                            \
+   compat_flush_signals(current);                                             \
+                                                                              \
+   daemonize();                                                               \
+   compat_reparent_to_init();                                                 \
+})
+#else
+#define compat_daemonize(x...) daemonize(x)
+#endif
+
+
+/*
+ * try to freeze a process. For kernels 2.6.11 or newer, we know how to choose
+ * the interface. The problem is that the oldest interface, introduced in
+ * 2.5.18, was backported to 2.4.x kernels. So if we're older than 2.6.11,
+ * we'll decide what to do based on whether or not swsusp was configured
+ * for the kernel.  For kernels 2.6.20 and newer, we'll also need to include
+ * freezer.h since the try_to_freeze definition was pulled out of sched.h.
+ */
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 20)
+#include <linux/freezer.h>
+#endif
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 13) || defined(VMW_TL10S64_WORKAROUND)
+#define compat_try_to_freeze() try_to_freeze()
+#elif LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 11)
+#define compat_try_to_freeze() try_to_freeze(PF_FREEZE)
+#elif defined(CONFIG_SOFTWARE_SUSPEND) || defined(CONFIG_SOFTWARE_SUSPEND2)
+#include "compat_mm.h"
+#include <linux/errno.h>
+#include <linux/suspend.h>
+static inline int compat_try_to_freeze(void)  { 
+   if (current->flags & PF_FREEZE) {
+      refrigerator(PF_FREEZE); 
+      return 1;
+   } else {
+      return 0;
+   }
+}
+#else
+static inline int compat_try_to_freeze(void) { return 0; }
+#endif
+
+/*
+ * As of 2.6.23-rc1, kernel threads are no longer freezable by
+ * default. Instead, kernel threads that need to be frozen must opt-in
+ * by calling set_freezable() as soon as the thread is created.
+ */
+
+#if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 22)
+#define compat_set_freezable() do { set_freezable(); } while (0)
+#else
+#define compat_set_freezable() do {} while (0)
+#endif
+
+/*
+ * Around 2.6.27 kernel stopped sending signals to kernel
+ * threads being frozen, instead threads have to check
+ * freezing() or use wait_event_freezable(). Unfortunately
+ * wait_event_freezable() completely hides the fact that
+ * thread was frozen from calling code and sometimes we do
+ * want to know that.
+ */
+#ifdef PF_FREEZER_NOSIG
+#define compat_wait_check_freezing() freezing(current)
+#else
+#define compat_wait_check_freezing() (0)
+#endif
+
+/*
+ * Since 2.6.27-rc2 kill_proc() is gone... Replacement (GPL-only!)
+ * API is available since 2.6.19.  Use them from 2.6.27-rc1 up.
+ */
+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 27)
+typedef int compat_pid;
+#define compat_find_get_pid(pid) (pid)
+#define compat_put_pid(pid) do { } while (0)
+#define compat_kill_pid(pid, sig, flag) kill_proc(pid, sig, flag)
+#else
+typedef struct pid * compat_pid;
+#define compat_find_get_pid(pid) find_get_pid(pid)
+#define compat_put_pid(pid) put_pid(pid)
+#define compat_kill_pid(pid, sig, flag) kill_pid(pid, sig, flag)
+#endif
+
+
+#endif /* __COMPAT_SCHED_H__ */

vmmon-only/include/compat_spinlock.h

@@ -0,0 +1,48 @@
+/*********************************************************
+ * Copyright (C) 2005 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef __COMPAT_SPINLOCK_H__
+#   define __COMPAT_SPINLOCK_H__
+
+#include <linux/spinlock.h>
+
+/*
+ * Preempt support was added during 2.5.x development cycle, and later
+ * it was backported to 2.4.x.  In 2.4.x backport these definitions
+ * live in linux/spinlock.h, that's why we put them here (in 2.6.x they
+ * are defined in linux/preempt.h which is included by linux/spinlock.h).
+ */
+#ifdef CONFIG_PREEMPT
+#define compat_preempt_disable() preempt_disable()
+#define compat_preempt_enable()  preempt_enable()
+#else
+#define compat_preempt_disable() do { } while (0)
+#define compat_preempt_enable()  do { } while (0)
+#endif
+
+/* Some older kernels - 2.6.10 and earlier - lack DEFINE_SPINLOCK */
+#ifndef DEFINE_SPINLOCK
+#define DEFINE_SPINLOCK(x) spinlock_t x = SPIN_LOCK_UNLOCKED
+#endif
+
+/* Same goes for DEFINE_RWLOCK */
+#ifndef DEFINE_RWLOCK
+#define DEFINE_RWLOCK(x)   rwlock_t x = RW_LOCK_UNLOCKED
+#endif
+
+#endif /* __COMPAT_SPINLOCK_H__ */

vmmon-only/include/compat_version.h

@@ -0,0 +1,131 @@
+/*********************************************************
+ * Copyright (C) 1998 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef __COMPAT_VERSION_H__
+#   define __COMPAT_VERSION_H__
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMKDRIVERS
+#include "includeCheck.h"
+
+
+#ifndef __linux__
+#   error "linux-version.h"
+#endif
+
+
+#include <linux/version.h>
+
+#ifndef KERNEL_VERSION
+#   error KERNEL_VERSION macro is not defined, environment is busted
+#endif
+
+
+/*
+ * Distinguish relevant classes of Linux kernels.
+ *
+ * The convention is that version X defines all
+ * the KERNEL_Y symbols where Y <= X.
+ *
+ * XXX Do not add more definitions here. This way of doing things does not
+ *     scale, and we are going to phase it out soon --hpreg
+ */
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 1, 0)
+#   define KERNEL_2_1
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 2, 0)
+#   define KERNEL_2_2
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 3, 1)
+#   define KERNEL_2_3_1
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 3, 15)
+/*   new networking */
+#   define KERNEL_2_3_15
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 3, 25)
+/*  new procfs */
+#   define KERNEL_2_3_25
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 3, 29)
+/*  even newer procfs */
+#   define KERNEL_2_3_29
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 3, 43)
+/*  softnet changes */
+#   define KERNEL_2_3_43
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 3, 47)
+/*  more softnet changes */
+#   define KERNEL_2_3_47
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 3, 99)
+/*  name in netdevice struct is array and not pointer */
+#   define KERNEL_2_3_99
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 4, 0)
+/*  New 'owner' member at the beginning of struct file_operations */
+#      define KERNEL_2_4_0
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 4, 8)
+/*  New netif_rx_ni() --hpreg */
+#   define KERNEL_2_4_8
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 5, 2)
+/*  New kdev_t, major()/minor() API --hpreg */
+#   define KERNEL_2_5_2
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 5, 5)
+/*  New sk_alloc(), pte_offset_map()/pte_unmap() --hpreg */
+#   define KERNEL_2_5_5
+#endif
+
+/* Linux kernel 3.0 can be called 2.6.40, and 3.1 can be 2.6.41...
+ * Use COMPAT_LINUX_VERSION_CHECK_LT iff you need to compare running kernel to
+ * versions 3.0 and above.
+ *
+ */
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 0, 0)
+   /* Straight forward comparison if kernel version is 3.0.0 and beyond */
+#   define COMPAT_LINUX_VERSION_CHECK_LT(a, b, c) LINUX_VERSION_CODE < KERNEL_VERSION (a, b, c)
+#elif LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 40)
+   /* Use b of the check to calculate corresponding c of kernel
+    *  version to compare */
+#   define COMPAT_LINUX_VERSION_CHECK_LT(a, b, c) LINUX_VERSION_CODE < KERNEL_VERSION (2, 6, (b + 40))
+#else
+    /* This is anyways lesser than any 3.x versions */
+#   define COMPAT_LINUX_VERSION_CHECK_LT(a, b, c) 1
+#endif
+
+#endif /* __COMPAT_VERSION_H__ */

vmmon-only/include/contextinfo.h

@@ -0,0 +1,80 @@
+/*********************************************************
+ * Copyright (C) 2005-2014,2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+
+/*
+ * contextinfo.h
+ *
+ *    Context structures shared across all products
+ */
+
+#ifndef _CONTEXTINFO_H
+#define _CONTEXTINFO_H
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMX
+#include "includeCheck.h"
+
+#include "x86desc.h"
+
+#pragma pack(push, 1)
+typedef struct Context64 {
+   uint64 cr3;
+   uint64 rax;
+   uint64 rcx;
+   uint64 rdx;
+   uint64 rbx;
+   uint64 rsi;
+   uint64 rdi;
+   uint64 rbp;
+   uint64 rsp;
+   uint64 r8;
+   uint64 r9;
+   uint64 r10;
+   uint64 r11;
+   uint64 r12;
+   uint64 r13;
+   uint64 r14;
+   uint64 r15;
+   uint32 cs;
+   uint32 ds;
+   uint32 ss;
+   uint32 es;
+   uint32 fs;
+   uint32 gs;
+   uint64 rip;
+   uint64 eflags;
+   uint16 ldt;
+   uint16 _pad[3];
+} Context64;
+#pragma pack(pop)
+
+#pragma pack(push, 1)
+typedef struct ContextInfo64 {
+   DTRWords64 gdtr;
+   DTRWords64 idtr;
+   Context64  context;
+   uint16     tr;
+   uint16     _pad0;
+} ContextInfo64;
+#pragma pack(pop)
+
+#endif

vmmon-only/include/cpu_defs.h

@@ -0,0 +1,49 @@
+/*********************************************************
+ * Copyright (C) 2009 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * cpu_defs.h --
+ *
+ *	CPU-related definitions shared between vmkernel and user-space.
+ */
+
+#ifndef CPU_DEFS_H
+#define CPU_DEFS_H
+
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMKDRIVERS
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+
+#define INCLUDE_ALLOW_VMX
+#include "includeCheck.h"
+
+#include "vm_basic_types.h"
+
+typedef uint32 PCPU;
+#define INVALID_PCPU		((PCPU) -1)
+
+#define MAX_PCPUS       1024
+#define MAX_PCPUS_BITS  10  // MAX_PCPUS <= (1 << MAX_PCPUS_BITS)
+#define MAX_PCPUS_MASK  ((1 << MAX_PCPUS_BITS) - 1)
+
+#endif

vmmon-only/include/crossgdt.h

@@ -0,0 +1,59 @@
+/*********************************************************
+ * Copyright (C) 2006-2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * crossgdt.h --
+ *
+ *      This GDT is used for switching between monitor and host context.
+ *      It contains the host and monitor basic segment descriptors.
+ *      There is only one in the whole host system, shared by all VMs.
+ *      It is allocated when the first VCPU is started and freed when the
+ *      driver is unloaded.
+ *
+ *      The crossGDT size is exactly one page.
+ *
+ *      The hosted world switch code is based on the assumption that by
+ *      placing VMM descriptors at the end of the page, they will not
+ *      overlap with host kernel descriptors in use when "crossing over".
+ *
+ *      All necessary host segments must be from first page of GDT.
+ *      In Nov 2006, host GDT limits easily met this constraint:
+ *
+ *          Linux 64 bit:  80 (yes 80, not 7F)
+ *          MacOS 64 bit:  8F
+ *        Windows 64 bit:  6F
+ */
+
+#ifndef _CROSSGDT_H_
+#define _CROSSGDT_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#include "vm_basic_defs.h"  // PAGE_SIZE
+#include "x86types.h"       // Descriptor
+
+
+typedef struct CrossGDT {
+   Descriptor gdtes[PAGE_SIZE / sizeof(Descriptor)];
+} CrossGDT;
+
+#endif

vmmon-only/include/driver-config.h

@@ -0,0 +1,79 @@
+/*********************************************************
+ * Copyright (C) 1998 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * Sets the proper defines from the Linux header files
+ *
+ * This file must be included before the inclusion of any kernel header file,
+ * with the exception of linux/autoconf.h and linux/version.h --hpreg
+ */
+
+#ifndef __VMX_CONFIG_H__
+#define __VMX_CONFIG_H__
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMKDRIVERS
+#include "includeCheck.h"
+
+#include "compat_version.h"
+#include "compat_autoconf.h"
+
+/*
+ * We rely on Kernel Module support.  Check here.
+ */
+#ifndef CONFIG_MODULES
+#   error "No Module support in this kernel.  Please configure with CONFIG_MODULES"
+#endif
+
+/*
+ * 2.2 kernels still use __SMP__ (derived from CONFIG_SMP
+ * in the main Makefile), so we do it here.
+ */
+
+#ifdef CONFIG_SMP
+#   define __SMP__ 1
+#endif
+
+#if defined(CONFIG_MODVERSIONS) && defined(KERNEL_2_1)
+#   if LINUX_VERSION_CODE < KERNEL_VERSION(2,5,60)
+/*
+ * MODVERSIONS might be already defined when using kernel's Makefiles.
+ */
+#      ifndef MODVERSIONS
+#         define MODVERSIONS
+#      endif
+#      include <linux/modversions.h>
+#   endif
+#endif
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 24)
+/*
+ * Force the uintptr_t definition to come from linux/types.h instead of vm_basic_types.h.
+ */
+#   include <linux/types.h>
+#   define _STDINT_H 1
+#endif
+
+#ifndef __KERNEL__
+#   define __KERNEL__
+#endif
+
+#endif

vmmon-only/include/includeCheck.h

@@ -0,0 +1,17 @@
+/*********************************************************
+ * Copyright (C) 2008,2018-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/

vmmon-only/include/iocontrols.h

@@ -0,0 +1,609 @@
+/*********************************************************
+ * Copyright (C) 1998-2020 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+
+/*
+ * iocontrols.h
+ *
+ *        The driver io controls.
+ */
+
+#ifndef _IOCONTROLS_H_
+#define _IOCONTROLS_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_MODULE
+#include "includeCheck.h"
+
+#ifdef VMX86_SERVER
+#error iocontrols.h is for hosted vmmon, do not use on visor
+#endif
+
+#include "x86segdescrs.h"
+#include "rateconv.h"
+#include "overheadmem_types.h"
+#include "pageLock_defs.h"
+#include "numa_defs.h"
+#include "bootstrap_vmm.h"
+#include "contextinfo.h"
+#include "vcpuid.h"
+#include "sharedAreaType.h"
+
+#if defined __cplusplus
+extern "C" {
+#endif
+
+
+/*
+ * Maximum number of shared regions that can be passed to MonLoader.
+ */
+#define ML_SHARED_REGIONS_MAX 3
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * VA64ToPtr --
+ *
+ *      Convert a VA64 to a pointer.
+ *
+ *      Usage of this function is strictly limited to these 2 cases:
+ *
+ *      1) In a VMX function which does an ioctl to vmmon, and receives a VMX
+ *         pointer as a result.
+ *
+ *      2) In the vmmon code, for the functions which have a VA64 and need
+ *         to call kernel APIs which take pointers.
+ *
+ * Results:
+ *      Virtual address.
+ *
+ * Side effects:
+ *      None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE void *
+VA64ToPtr(VA64 va64) // IN
+{
+#ifdef VM_64BIT
+   ASSERT_ON_COMPILE(sizeof (void *) == 8);
+#else
+   ASSERT_ON_COMPILE(sizeof (void *) == 4);
+   // Check that nothing of value will be lost.
+   ASSERT(!(va64 >> 32));
+#endif
+   return (void *)(uintptr_t)va64;
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * PtrToVA64 --
+ *
+ *      Convert a pointer to a VA64.
+ *
+ *      Usage of this function is strictly limited to these 2 cases:
+ *
+ *      1) In a VMX function which does an ioctl to vmmon, and passes in a VMX
+ *         pointer.
+ *
+ *      2) In the vmmon code, for the functions which need to pass in a kernel
+ *         pointer to functions which can take either a user or a kernel
+ *         pointer in the same parameter.
+ *
+ * Results:
+ *      Virtual address.
+ *
+ * Side effects:
+ *      None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE VA64
+PtrToVA64(void const *ptr) // IN
+{
+   ASSERT_ON_COMPILE(sizeof ptr <= sizeof (VA64));
+   return (VA64)(uintptr_t)ptr;
+}
+
+
+/*
+ * Driver version.
+ *
+ * Increment major version when you make an incompatible change.
+ * Compatibility goes both ways (old driver with new executable
+ * as well as new driver with old executable).
+ *
+ * Note: Vmcore compatibility is different from driver versioning.
+ * For vmcore puposes, the bora tree is conceptually split in two:
+ * vmcore, and rest-of-bora. The vmmon driver is largely outside
+ * vmcore and vmcore imports functionality from vmmon. Addition,
+ * deletion or modification of an iocontrol used only by rest-of-bora
+ * does not break vmcore compatibility.
+ *
+ * See bora/doc/vmcore details.
+ *
+ * If you increment VMMON_VERSION, you must also increment
+ * the NT specific VMX86_DRIVER_VERSION.
+ */
+
+#define VMMON_VERSION           (392 << 16 | 0)
+#define VMMON_VERSION_MAJOR(v)  ((uint32) (v) >> 16)
+#define VMMON_VERSION_MINOR(v)  ((uint16) (v))
+
+
+/*
+ * ENOMEM returned after MAX_VMS virtual machines created
+ */
+
+#define MAX_VMS 64
+
+/*
+ * MsgWaitForMultipleObjects doesn't scale well enough on Win32.
+ * Allocate with MAX_VMS so static buffers are large, but do
+ * admissions control with this value on Win32 until we check
+ * scalability (probably in authd).
+ */
+#ifdef _WIN32
+#define MAX_VMS_WIN32 64
+#endif
+
+
+#if !__linux__
+/*
+ * On platforms other than Linux, IOCTLCMD_foo values are just numbers, and
+ * we build the IOCTL_VMX86_foo values around these using platform-specific
+ * format for encoding arguments and sizes.
+ */
+#  define IOCTLCMD(_cmd) IOCTLCMD_ ## _cmd
+#else // if __linux__
+/*
+ * Linux defines _IO* macros, but the core kernel code ignore the encoded
+ * ioctl value. It is up to individual drivers to decode the value (for
+ * example to look at the size of a structure to determine which version
+ * of a specific command should be used) or not (which is what we
+ * currently do, so right now the ioctl value for a given command is the
+ * command itself).
+ *
+ * Hence, we just define the IOCTL_VMX86_foo values directly, with no
+ * intermediate IOCTLCMD_ representation.
+ */
+#  define IOCTLCMD(_cmd) IOCTL_VMX86_ ## _cmd
+#endif
+
+
+enum IOCTLCmd {
+   /*
+    * We need to bracket the range of values used for ioctls, because x86_64
+    * Linux forces us to explicitly register ioctl handlers by value for
+    * handling 32 bit ioctl syscalls.  Hence FIRST and LAST.  FIRST must be
+    * 2001 so that VERSION is 2001 for backwards compatibility.
+    */
+#if defined __linux__ || defined _WIN32
+   /* Start at 2001 because legacy code did. */
+   IOCTLCMD(FIRST) = 2001,
+#else
+   /* Start at 0. */
+   IOCTLCMD(FIRST),
+#endif
+   IOCTLCMD(VERSION) = IOCTLCMD(FIRST),
+   IOCTLCMD(CREATE_VM),
+   IOCTLCMD(PROCESS_BOOTSTRAP),
+   IOCTLCMD(REGISTER_SHARED),
+   IOCTLCMD(REGISTER_STATVARS),
+   IOCTLCMD(RELEASE_VM),
+   IOCTLCMD(GET_NUM_VMS),
+   IOCTLCMD(RUN_VM),
+   IOCTLCMD(LOOK_UP_MPN),
+   IOCTLCMD(GET_VMM_PAGE_ROOT),
+   IOCTLCMD(LOCK_PAGE),
+   IOCTLCMD(UNLOCK_PAGE),
+   IOCTLCMD(APIC_INIT),
+   IOCTLCMD(SET_HARD_LIMIT),
+   IOCTLCMD(GET_MEM_INFO),
+   IOCTLCMD(ADMIT),
+   IOCTLCMD(UPDATE_MEM_INFO),
+   IOCTLCMD(READMIT),
+   IOCTLCMD(GET_KHZ_ESTIMATE),
+   IOCTLCMD(SET_HOST_CLOCK_RATE),
+   IOCTLCMD(READ_PAGE),
+   IOCTLCMD(WRITE_PAGE),
+   IOCTLCMD(LOCK_PAGE_NEW),
+   IOCTLCMD(UNLOCK_PAGE_BY_MPN),
+    /* AWE calls */
+   IOCTLCMD(ALLOC_LOCKED_PAGES),
+   IOCTLCMD(GET_NEXT_ANON_PAGE),
+   IOCTLCMD(GET_NUM_ANON_PAGES),
+
+   IOCTLCMD(GET_ALL_MSRS),
+
+   IOCTLCMD(GET_REFERENCE_CLOCK_HZ),
+   IOCTLCMD(INIT_PSEUDO_TSC),
+   IOCTLCMD(CHECK_PSEUDO_TSC),
+   IOCTLCMD(GET_PSEUDO_TSC),
+
+   IOCTLCMD(SYNC_GET_TSCS),
+
+   IOCTLCMD(GET_IPI_VECTORS),
+   IOCTLCMD(SEND_IPI),
+   IOCTLCMD(SEND_ONE_IPI),
+
+   /*
+    * Keep host-specific calls at the end so they can be undefined
+    * without renumbering the common calls.
+    */
+
+#if defined __linux__
+   IOCTLCMD(SET_UID),		// VMX86_DEVEL only
+   IOCTLCMD(GET_ALL_CPUID),
+#endif
+
+#if defined _WIN32
+   IOCTLCMD(HARD_LIMIT_MONITOR_STATUS), // used by vmauthd on Windows
+   IOCTLCMD(CHANGE_HARD_LIMIT),         // used by vmauthd on Windows
+   IOCTLCMD(READ_DISASM_PROC_BINARY),
+   IOCTLCMD(CHECK_CANDIDATE_VA64),
+   IOCTLCMD(SET_MEMORY_PARAMS),
+   IOCTLCMD(REMEMBER_KHZ_ESTIMATE),
+   IOCTLCMD(REMAP_SCATTER_LIST),
+   IOCTLCMD(REMAP_SCATTER_LIST_RO),     // map the list as read-only
+   IOCTLCMD(UNMAP_SCATTER_LIST),
+#endif
+
+#if defined __APPLE__
+   IOCTLCMD(GET_NUM_RESPONDING_CPUS),
+   IOCTLCMD(INIT_DRIVER),
+   IOCTLCMD(BLUEPILL),
+#endif
+
+   IOCTLCMD(GET_UNAVAIL_PERF_CTRS),
+   IOCTLCMD(GET_MONITOR_CONTEXT),
+   // Must be last.
+   IOCTLCMD(LAST)
+};
+
+
+#if defined _WIN32
+/*
+ * Windows ioctl definitions.
+ *
+ * We use the IRP Information field for the return value
+ * of IOCTLCMD_RUN_VM, to be faster since it is used a lot.
+ */
+
+#define FILE_DEVICE_VMX86        0x8101
+#define VMX86_IOCTL_BASE_INDEX   0x801
+#define VMIOCTL_BUFFERED(name) \
+     CTL_CODE(FILE_DEVICE_VMX86, \
+	       VMX86_IOCTL_BASE_INDEX + IOCTLCMD_ ## name, \
+	       METHOD_BUFFERED, \
+	       FILE_ANY_ACCESS)
+#define VMIOCTL_NEITHER(name) \
+      CTL_CODE(FILE_DEVICE_VMX86, \
+	       VMX86_IOCTL_BASE_INDEX + IOCTLCMD_ ## name, \
+	       METHOD_NEITHER, \
+	       FILE_ANY_ACCESS)
+
+#define IOCTL_VMX86_VERSION             VMIOCTL_BUFFERED(VERSION)
+#define IOCTL_VMX86_CREATE_VM           VMIOCTL_BUFFERED(CREATE_VM)
+#define IOCTL_VMX86_RELEASE_VM          VMIOCTL_BUFFERED(RELEASE_VM)
+#define IOCTL_VMX86_PROCESS_BOOTSTRAP   VMIOCTL_BUFFERED(PROCESS_BOOTSTRAP)
+#define IOCTL_VMX86_REGISTER_SHARED     VMIOCTL_BUFFERED(REGISTER_SHARED)
+#define IOCTL_VMX86_REGISTER_STATVARS   VMIOCTL_BUFFERED(REGISTER_STATVARS)
+#define IOCTL_VMX86_GET_NUM_VMS         VMIOCTL_BUFFERED(GET_NUM_VMS)
+#define IOCTL_VMX86_RUN_VM              VMIOCTL_NEITHER(RUN_VM)
+#define IOCTL_VMX86_SEND_IPI            VMIOCTL_NEITHER(SEND_IPI)
+#define IOCTL_VMX86_SEND_ONE_IPI        VMIOCTL_BUFFERED(SEND_ONE_IPI)
+#define IOCTL_VMX86_GET_IPI_VECTORS     VMIOCTL_BUFFERED(GET_IPI_VECTORS)
+#define IOCTL_VMX86_LOOK_UP_MPN         VMIOCTL_BUFFERED(LOOK_UP_MPN)
+#define IOCTL_VMX86_GET_VMM_PAGE_ROOT   VMIOCTL_BUFFERED(GET_VMM_PAGE_ROOT)
+#define IOCTL_VMX86_LOCK_PAGE           VMIOCTL_BUFFERED(LOCK_PAGE)
+#define IOCTL_VMX86_UNLOCK_PAGE         VMIOCTL_BUFFERED(UNLOCK_PAGE)
+#define IOCTL_VMX86_APIC_INIT           VMIOCTL_BUFFERED(APIC_INIT)
+#define IOCTL_VMX86_SET_HARD_LIMIT      VMIOCTL_BUFFERED(SET_HARD_LIMIT)
+#define IOCTL_VMX86_GET_MEM_INFO        VMIOCTL_BUFFERED(GET_MEM_INFO)
+#define IOCTL_VMX86_ADMIT               VMIOCTL_BUFFERED(ADMIT)
+#define IOCTL_VMX86_READMIT             VMIOCTL_BUFFERED(READMIT)
+#define IOCTL_VMX86_UPDATE_MEM_INFO     VMIOCTL_BUFFERED(UPDATE_MEM_INFO)
+#define IOCTL_VMX86_HARD_LIMIT_MONITOR_STATUS   VMIOCTL_BUFFERED(HARD_LIMIT_MONITOR_STATUS)
+#define IOCTL_VMX86_CHANGE_HARD_LIMIT   VMIOCTL_BUFFERED(CHANGE_HARD_LIMIT)
+
+#define IOCTL_VMX86_GET_TOTAL_MEM_USAGE	VMIOCTL_BUFFERED(GET_TOTAL_MEM_USAGE)
+#define IOCTL_VMX86_GET_KHZ_ESTIMATE    VMIOCTL_BUFFERED(GET_KHZ_ESTIMATE)
+#define IOCTL_VMX86_SET_HOST_CLOCK_RATE VMIOCTL_BUFFERED(SET_HOST_CLOCK_RATE)
+#define IOCTL_VMX86_SYNC_GET_TSCS       VMIOCTL_BUFFERED(SYNC_GET_TSCS)
+#define IOCTL_VMX86_READ_PAGE           VMIOCTL_BUFFERED(READ_PAGE)
+#define IOCTL_VMX86_WRITE_PAGE          VMIOCTL_BUFFERED(WRITE_PAGE)
+#define IOCTL_VMX86_LOCK_PAGE_NEW       VMIOCTL_BUFFERED(LOCK_PAGE_NEW)
+#define IOCTL_VMX86_UNLOCK_PAGE_BY_MPN  VMIOCTL_BUFFERED(UNLOCK_PAGE_BY_MPN)
+#define IOCTL_VMX86_ALLOC_LOCKED_PAGES  VMIOCTL_BUFFERED(ALLOC_LOCKED_PAGES)
+#define IOCTL_VMX86_GET_NEXT_ANON_PAGE  VMIOCTL_BUFFERED(GET_NEXT_ANON_PAGE)
+#define IOCTL_VMX86_GET_NUM_ANON_PAGES  VMIOCTL_BUFFERED(GET_NUM_ANON_PAGES)
+
+#define IOCTL_VMX86_READ_DISASM_PROC_BINARY \
+                                      VMIOCTL_BUFFERED(READ_DISASM_PROC_BINARY)
+#define IOCTL_VMX86_CHECK_CANDIDATE_VA64 VMIOCTL_BUFFERED(CHECK_CANDIDATE_VA64)
+
+#define IOCTL_VMX86_SET_MEMORY_PARAMS   VMIOCTL_BUFFERED(SET_MEMORY_PARAMS)
+
+#define IOCTL_VMX86_REMEMBER_KHZ_ESTIMATE VMIOCTL_BUFFERED(REMEMBER_KHZ_ESTIMATE)
+
+#define IOCTL_VMX86_GET_ALL_MSRS        VMIOCTL_BUFFERED(GET_ALL_MSRS)
+
+#define IOCTL_VMX86_GET_REFERENCE_CLOCK_HZ   VMIOCTL_BUFFERED(GET_REFERENCE_CLOCK_HZ)
+#define IOCTL_VMX86_INIT_PSEUDO_TSC          VMIOCTL_BUFFERED(INIT_PSEUDO_TSC)
+#define IOCTL_VMX86_CHECK_PSEUDO_TSC         VMIOCTL_BUFFERED(CHECK_PSEUDO_TSC)
+#define IOCTL_VMX86_GET_PSEUDO_TSC           VMIOCTL_NEITHER(GET_PSEUDO_TSC)
+#define IOCTL_VMX86_GET_UNAVAIL_PERF_CTRS    VMIOCTL_NEITHER(GET_UNAVAIL_PERF_CTRS)
+#define IOCTL_VMX86_GET_MONITOR_CONTEXT      VMIOCTL_BUFFERED(GET_MONITOR_CONTEXT)
+#define IOCTL_VMX86_REMAP_SCATTER_LIST    VMIOCTL_BUFFERED(REMAP_SCATTER_LIST)
+#define IOCTL_VMX86_REMAP_SCATTER_LIST_RO VMIOCTL_BUFFERED(REMAP_SCATTER_LIST_RO)
+#define IOCTL_VMX86_UNMAP_SCATTER_LIST    VMIOCTL_BUFFERED(UNMAP_SCATTER_LIST)
+#endif
+
+
+#define INIT_BLOCK_MAGIC     (0x1789 + 14)
+
+
+typedef
+#include "vmware_pack_begin.h"
+struct VMLockPageRet {
+   MPN   mpn;      // OUT: MPN
+   int32 status;   // OUT: PAGE_* status code
+}
+#include "vmware_pack_end.h"
+VMLockPageRet;
+
+typedef
+#include "vmware_pack_begin.h"
+union {
+   VA64 uAddr;        // IN: user address
+   VMLockPageRet ret; // OUT: status code and MPN
+}
+#include "vmware_pack_end.h"
+VMLockPage;
+
+typedef
+#include "vmware_pack_begin.h"
+union {
+   Vcpuid vcpuid; // IN: VCPU
+   MPN pageRoot;  // OUT: MPN of the VCPU's page root
+}
+#include "vmware_pack_end.h"
+VcpuPageRoot;
+
+#define VMX86_DRIVER_VCPUID_OFFSET 1000
+
+
+/*
+ * We keep track of 3 different limits on the number of pages we can lock.
+ * The host limit is determined at driver load time (in windows only) to
+ * make sure we do not starve the host by locking too many pages.
+ * The static limit is user defined in the UI and the dynamic limit is
+ * set by authd's hardLimitMonitor code (windows only), which queries
+ * host load and adjusts the limit accordingly.  We lock the minimum of
+ * all these values at any given time.
+ */
+typedef struct LockedPageLimit {
+   PageCnt host;        // driver calculated maximum for this host
+   PageCnt configured;  // user defined maximum pages to lock
+   PageCnt dynamic;     // authd hardLimitMonitor pages to lock
+} LockedPageLimit;
+
+/*
+ * Sentinel VA for IOCTL_VMX86_SET_MEMORY_PARAMS, indicates
+ * NtQuerySystemInformation should be used to determine the host
+ * LockedPageLimit.
+ */
+#define MEMORY_PARAM_USE_SYSINFO_FOR_LOCKED_PAGE_LIMIT   ((VA64)(int64)-1)
+
+/*
+ * Data structures for the GET_MEM_INFO and ADMIT ioctls.
+ *
+ * Be careful adding structs and fields to VMMemInfoArgs and its
+ * substructures. These are compiled into both the 32-bit and 64-bit
+ * vmmon drivers and the 32-bit and 64-bit vmx's and need to have
+ * the same size and layout in all four combinations. Note the
+ * use of padding below to ensure that this happens.
+ */
+
+typedef struct VMMemMgmtInfo {
+   PageCnt         minAllocation;   // minimum pages for vm
+   PageCnt         maxAllocation;   // maximum pages the vm could lock
+   PageCnt         nonpaged;        // overhead memory (guest, mmap)
+   PageCnt         paged;           // vmx memory (malloc, statics)
+   PageCnt         anonymous;       // vmm memory
+   PageCnt         mainMemSize;     // guest main memory size
+   PageCnt         locked;          // number of pages locked by this vm
+   PageCnt         perVMOverhead;   // memory for vmx/vmmon overheads
+   uint32          shares;          // proportional sharing weight
+   Percent         touchedPct;      // % of guest memory being touched
+   Percent         dirtiedPct;      // % of guest memory being dirtied
+   Bool            admitted;        // admission control
+   uint8           _pad;            // for alignment of 64-bit fields
+   uint64          hugePageBytes;   // number of bytes occupied by huge pages
+   uint64          timestamp;       // most recent poll of get mem info time
+} VMMemMgmtInfo;
+
+typedef struct VMMemMgmtInfoPatch {
+   Percent         touchedPct;      // % of guest memory being touched
+   Percent         dirtiedPct;      // % of guest memory being dirtied
+   uint8           _pad[6];
+   uint64          hugePageBytes;
+} VMMemMgmtInfoPatch;
+
+/*
+ * See comment on padding and size/layout constraints above when
+ * when modifying VMMemInfoArgs or its components.
+ */
+
+typedef struct VMMemInfoArgs {
+   uint64          currentTime;        // Host time in secs of the call.
+   PageCnt         globalMinAllocation;// pages that must fit in maxLockedPages
+   PageCnt         numLockedPages;     // total locked pages by all vms
+   LockedPageLimit lockedPageLimit;    // set of locked page limits
+   PageCnt         maxLockedPages;     // effective limit on locked pages
+   uint32          callerIndex;        // this vm's index memInfo array
+   uint32          numVMs;             // number of running VMs
+   Percent         minVmMemPct;        // % of vm that must fit in memory
+   uint8           _pad[7];
+   VMMemMgmtInfo   memInfo[1];
+} VMMemInfoArgs;
+
+#define VM_GET_MEM_INFO_SIZE(numVMs) \
+   (sizeof(VMMemInfoArgs) - sizeof(VMMemMgmtInfo) + (numVMs) * sizeof(VMMemMgmtInfo))
+
+typedef struct VMMPNNext {
+   MPN         inMPN;   // IN
+   MPN         outMPN;  // OUT
+} VMMPNNext;
+
+typedef struct VMMPNList {
+   PageCnt   mpnCount;   // IN (and OUT on Mac OS)
+   Bool      ignoreLimits;
+   uint8     _pad[7];
+   VA64      mpnList;    // IN: User VA of an array of 64-bit MPNs.
+} VMMPNList;
+
+typedef struct VMMUnlockPageByMPN {
+   MPN       mpn;
+   VA64      uAddr;         /* IN: User VA of the page (optional). */
+} VMMUnlockPageByMPN;
+
+typedef struct VMMReadWritePage {
+   MPN          mpn;   // IN
+   VA64         uAddr; // IN: User VA of a PAGE_SIZE-large buffer.
+} VMMReadWritePage;
+
+/*
+ * Data structure for the INIT_PSEUDO_TSC and CHECK_PSEUDO_TSC.
+ */
+
+typedef struct PTSCInitParams {
+   RateConv_Params refClockToPTSC;
+   uint64          tscHz;
+   uint64          initialPTSC;
+   int64           tscOffset;
+   Bool            forceRefClock;
+   Bool            forceTSC;
+   Bool            hwTSCsSynced;
+   uint8           _pad[5];
+} PTSCInitParams;
+
+typedef struct PTSCCheckParams {
+   uint64 lastTSC;
+   uint64 lastRC;
+   Bool   usingRefClock;
+   uint8  _pad[7];
+} PTSCCheckParams;
+
+typedef struct IPIVectors {
+   /*
+    * Vectors we have allocated or stolen for the monitor interrupts.
+    */
+   uint8 monitorIPIVector;
+   uint8 hvIPIVector;
+} IPIVectors;
+
+/*
+ * Arguments and return value for VM creation.
+ */
+typedef struct VMCreateBlock {
+   VA64               bsBlob;        // IN: User VA of the VMM bootstrap blob.
+   uint32             bsBlobSize;    // IN: Size of VMM bootstrap blob.
+   uint32             numVCPUs;      // IN: Number of VCPUs.
+   uint16             vmid;          // OUT: VM ID for the created VM.
+} VMCreateBlock;
+
+/*
+ * Information about a shared region.
+ */
+typedef struct VMSharedRegion {
+   SharedAreaType index;
+   VPN            baseVpn;
+   uint32         numPages;
+} VMSharedRegion;
+
+/*
+ * Arguments for VMM shared area registration.
+ */
+typedef struct VMSharedAreaRegistrationBlock {
+   Vcpuid vcpu;            // IN: VCPU being registered.
+   VMSharedRegion region;  // IN: Shared region being registered.
+} VMSharedAreaRegistrationBlock;
+
+/*
+ * Information about a VM's statvars.
+ */
+typedef struct VMStatVarsRegistrationBlock {
+   VPN     baseVpn;
+   PageCnt numPages;
+   Vcpuid  vcpu;
+} VMStatVarsRegistrationBlock;
+
+typedef struct {
+   VA64   crosspage; // IN: User VA of VCPU crosspage.
+   VA64   ptRoot;    // IN: User VA of VCPU L4 page table root.
+} PerVcpuPages;
+
+/*
+ * Arguments for VMM bootstrap processing.
+ */
+typedef struct VMProcessBootstrapBlock {
+   VA64           bsBlobAddr;    // IN: User VA of the VMM bootstrap blob.
+   uint32         numBytes;      // IN: Size of VMM bootstrap blob.
+   uint32         headerOffset;  // IN: Offset of header in blob.
+   uint16         numVCPUs;      // IN: Number of VCPUs.
+   VMSharedRegion shRegions[ML_SHARED_REGIONS_MAX]; // IN: Shared regions.
+   PerVcpuPages   perVcpuPages[0];
+} VMProcessBootstrapBlock;
+
+static INLINE size_t
+GetVMProcessBootstrapBlockSize(unsigned numVCPUs)
+{
+   VMProcessBootstrapBlock *args = NULL;
+   return sizeof *args + numVCPUs * sizeof args->perVcpuPages[0];
+}
+
+/*
+ * Arguments for VMM context retrieval.
+ */
+typedef union {
+   Vcpuid vcpuid;     // IN
+   Context64 context; // OUT
+} VMMonContext;
+
+#if defined __APPLE__
+#   include "iocontrolsMacos.h"
+#endif
+
+/* Clean up helper macros */
+#undef IOCTLCMD
+
+
+#if defined __cplusplus
+} // extern "C"
+#endif
+
+#endif // ifndef _IOCONTROLS_H_

vmmon-only/include/memDefaults.h

@@ -0,0 +1,161 @@
+/*********************************************************
+ * Copyright (C) 1998-2016 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+
+#ifndef _MEMDEFAULTS_H_
+#define _MEMDEFAULTS_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_MODULE
+#include "includeCheck.h"
+
+#include "vm_basic_math.h"
+#include "vm_basic_defs.h"
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+#define MEMDEFAULTS_MIN_HOST_PAGES   MBYTES_2_PAGES(128)
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * MemDefaults_CalcMaxLockedPages --
+ *
+ *      Calculate the rough estimate of the maximum amount of memory
+ *      that can be locked (total for the kernel, all VMs, and other apps),
+ *      based on the size of host memory as supplied in pages.
+ *
+ * Results:
+ *      The estimated maximum memory that can be locked in pages.
+ *
+ * Side effects:
+ *      None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE unsigned
+MemDefaults_CalcMaxLockedPages(unsigned hostPages)  // IN:
+{
+   unsigned reservedPages;
+
+#if defined(__APPLE__)
+   /*
+    * Reserve (25% of the host memory + 512 MB) or 4 GB, whichever is lower.
+    * 4 GB hosts perform poorly with less than 1.5 GB reserved, and large
+    * memory hosts (>= 16 GB) may want to use more than 75% for VMs.
+    */
+   reservedPages = MIN((hostPages / 4) + MBYTES_2_PAGES(512),
+                       GBYTES_2_PAGES(4));
+#elif defined(_WIN32)
+   {
+      unsigned int hostGig = PAGES_2_GBYTES(hostPages);
+
+      if (hostGig <= 4) {
+         reservedPages = hostPages / 4;
+      } else if (hostGig >= 16) {
+         reservedPages = hostPages / 8;
+      } else {
+         /*
+          * Increment by 1/32 for each 4GB of host mem between 4 and 16.
+          * See PR779556.
+          */
+         reservedPages = hostPages / 32 * (8 - hostGig / 4);
+      }
+   }
+#else  // Linux
+   reservedPages = hostPages / 8;
+#endif
+
+   reservedPages = MAX(reservedPages, MEMDEFAULTS_MIN_HOST_PAGES);
+
+   return hostPages > reservedPages ? hostPages - reservedPages : 0;
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * MemDefaults_CalcMaxLockedMBs --
+ *
+ *      Calculate the rough estimate of the maximum amount of memory
+ *      that can be locked based on the size of host memory as supplied
+ *      in MBytes.
+ *
+ * Results:
+ *      The estimated maximum memory that can be locked in MBytes.
+ *
+ * Side effects:
+ *      None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE uint32
+MemDefaults_CalcMaxLockedMBs(uint32 hostMem)  // IN:
+{
+   return PAGES_2_MBYTES(
+             MemDefaults_CalcMaxLockedPages(MBYTES_2_PAGES(hostMem)));
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * MemDefaults_CalcMinReservedMBs --
+ *
+ *      Provide a lower bound on the user as to the minimum amount
+ *      of memory to lock based on the size of host memory. This
+ *      threshold might be crossed as a result of the user limiting
+ *      the amount of memory consumed by all VMs.
+ *
+ * Results:
+ *      The minimum locked memory requirement in MBytes.
+ *
+ * Side effects:
+ *      None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE uint32
+MemDefaults_CalcMinReservedMBs(uint32 hostMem)  // IN:
+{
+   if (hostMem < 512) {
+      return 32;
+   } else if (hostMem < 1024) {
+      return 64;
+   } else {
+      return 128;
+   }
+}
+
+
+void MemDefaults_GetReservedMemory(uint32 *host, uint32 *min,
+                                   uint32 *max, uint32 *recommended);
+
+#if defined(__cplusplus)
+}  // extern "C"
+#endif
+
+#endif

vmmon-only/include/modulecall.h

@@ -0,0 +1,469 @@
+/*********************************************************
+ * Copyright (C) 1998-2020 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+
+/*
+ * modulecall.h
+ *
+ *        Monitor <-->  Module (kernel driver) interface
+ */
+
+#ifndef _MODULECALL_H
+#define _MODULECALL_H
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+#include "x86types.h"
+#include "x86desc.h"
+#include "ptsc.h"
+#include "vcpuid.h"
+#include "vcpuset.h"
+#include "vmm_constants.h"
+#include "contextinfo.h"
+#include "rateconv.h"
+#include "modulecallstructs.h"
+#include "mon_assert.h"
+
+#define NUM_EXCEPTIONS   20     /* EXC_DE ... EXC_XF. */
+#define CP_STUB_SIZE     16     /* A relative jmp instruction (5 bytes)
+                                   padded to the next 16 byte boundary. */
+
+#define MODULECALL_TABLE                                                      \
+   MC(INTR)                                                                   \
+   MC(SEMAWAIT)                                                               \
+   MC(SEMASIGNAL)                                                             \
+   MC(SEMAFORCEWAKEUP)                                                        \
+   MC(IPI)          /* Hit multiple threads with IPI. */                      \
+   MC(ONE_IPI)      /* Hit a single thread with IPI. */                       \
+   MC(USERRETURN)   /* Return codes for user calls. */                        \
+   MC(GET_RECYCLED_PAGES)                                                     \
+   MC(RELEASE_ANON_PAGES)                                                     \
+   MC(LOOKUP_MPN)                                                             \
+   MC(COSCHED)                                                                \
+   MC(ALLOC_CONTIG_PAGES)                                                     \
+   MC(ALLOC_VMX_PAGE)                                                         \
+   MC(ALLOC_TMP_GDT)                                                          \
+   MC(PIN_MPN)                                                                \
+   MC(VMCLEAR_VMCS_ALL_CPUS)                                                  \
+   MC(GET_PAGE_ROOT)                                                          \
+   MC(ALLOC_ANON_LOW_PAGE)                                                    \
+   MC(GET_MON_IPI_VECTOR)                                                     \
+   MC(GET_HV_IPI_VECTOR)                                                      \
+   MC(GET_HOST_TIMER_VECTORS)                                                 \
+   MC(BOOTSTRAP_CLEANUP)                                                      \
+   MC(GET_SHARED_AREA)                                                        \
+   MC(GET_STAT_VARS)                                                          \
+   MC(GET_NUM_PTP_PAGES)                                                      \
+   MC(GET_HV_IO_BITMAP)                                                       \
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * ModuleCallType --
+ *
+ *      Enumeration of support calls done by the module.
+ *
+ *----------------------------------------------------------------------
+ */
+
+typedef enum ModuleCallType {
+   MODULECALL_NONE = 100,
+#define MC(_modulecall) MODULECALL_##_modulecall,
+   MODULECALL_TABLE
+#undef MC
+   MODULECALL_LAST                   // Number of entries. Must be the last one
+} ModuleCallType;
+
+#define MODULECALL_USERCALL_NONE     300
+
+/*
+ * Define VMX86_UCCOST in the makefiles (Local.mk,
+ * typically) if you want a special build whose only purpose
+ * is to measure the overhead of a user call and its
+ * breakdown.
+ *
+ * WINDOWS NOTE: I don't know how to pass VMX86_UCCOST to
+ * the driver build on Windows.  It must be defined by hand.
+ *
+ * ESX Note: we don't have a crosspage in which to store these
+ * timestamps.  Such a feature would perhaps be nice (if we
+ * ever tire of the argument that esx does so few usercalls
+ * that speed doesn't matter).
+ */
+
+#if defined(VMX86_UCCOST) && !defined(VMX86_SERVER)
+#define UCTIMESTAMP(cp, stamp) \
+             do { (cp)->ucTimeStamps[UCCOST_ ## stamp] = RDTSC(); } while (0)
+#else
+#define UCTIMESTAMP(cp, stamp)
+#endif
+
+#ifdef VMX86_SERVER
+typedef struct UCCostResults {
+   uint32 vmksti;
+   uint32 vmkcli;
+   uint32 ucnop;
+} UCCostResults;
+#else
+
+typedef struct UCCostResults {
+   uint32 htom;
+   uint32 mtoh;
+   uint32 ucnop;
+} UCCostResults;
+
+typedef enum UCCostStamp {
+#define UC(x) UCCOST_ ## x,
+#include "uccostTable.h"
+   UCCOST_MAX
+} UCCostStamp;
+#endif // VMX86_SERVER
+
+#define SHADOW_DR(cpData, n)    (cpData)->shadowDR[n].ureg64
+
+
+/*----------------------------------------------------------------------
+ *
+ * MAX_SWITCH_PT_PATCHES
+ *
+ *   This is the maximum number of patches that must be placed into
+ *   the monitor page tables so that the host GDT and the crosspage
+ *   can be accessed during worldswitch.
+ *
+ *----------------------------------------------------------------------
+ */
+#define MAX_SWITCH_PT_PATCHES 2
+
+/*----------------------------------------------------------------------
+ *
+ * WS_INTR_STRESS
+ *
+ *   When set to non-zero, world switch code will enable single-step
+ *   debugging across much of the switch path in both directions.  The
+ *   #DB handler detects single-stepping and induces a simulated NMI per
+ *   instruction.  This verifies that interrupts and exceptions are safe
+ *   across the switch path, even if an NMI were raised during handling
+ *   of another exception.
+ *
+ *   When set to zero, normal worldswitch operation occurs.
+ *
+ *   See the worldswitch assembly code for details.
+ *
+ *----------------------------------------------------------------------
+ */
+#define WS_INTR_STRESS 0
+
+
+/*----------------------------------------------------------------------
+ *
+ * VMMPageTablePatch
+ *
+ *    Describes an entry in the monitor page table which needs to be
+ *    patched during the back-to-host worldswitch.
+ *
+ *    o A patch can appear at any place in the page table, and so four
+ *      items are required to uniquely describe the patch:
+ *
+ *      o level
+ *
+ *        This is the level in the page table to which the patch must
+ *        be applied: L4, L3, L2, L1.  This information is used to
+ *        determine the base of the region of memory which must be
+ *        patched. The value zero is reserved to indicate an empty spot
+ *        in the array of patches.
+ *
+ *      o index
+ *
+ *        The index of the PTE at the given page table.
+ *
+ *      o ptIdx
+ *
+ *        The index of the page table at the given level.
+ *
+ *      o pte
+ *
+ *        This is the PTE value which will be patched into the monitor
+ *        page table.
+ *
+ *----------------------------------------------------------------------
+ */
+typedef
+#include "vmware_pack_begin.h"
+struct VMMPageTablePatch {
+#define PTP_EMPTY    (0U) /* Unused array entry. (must be 0) */
+#define PTP_LEVEL_L1 (1U)
+#define PTP_LEVEL_L2 (2U)
+#define PTP_LEVEL_L3 (3U)
+#define PTP_LEVEL_L4 (4U)       /* root level */
+   uint32   level;              /* [0, 4]  (maximal size: 3 bits) */
+   uint32   pteIdx;             /* Index of the PTE in the page table. */
+   uint64   pteGlobalIdx;       /* Global index of the PTE in 'level'. */
+   LPN      lpn;                /* Logical page number mapped by patch. */
+   VM_PDPTE pte;                /* PTE.                                */
+}
+#include "vmware_pack_end.h"
+VMMPageTablePatch;
+
+#define MODULECALL_NUM_ARGS  4
+
+/*
+ * The cross page contains tiny stacks upon which interrupt and exception
+ * handlers in the switch path may temporarily run.  Each stack must be
+ * large enough for the sum of:
+ *
+ * - 1 #DB exception frame (5 * uint64)
+ * - 1 #NMI exception frame (5 * uint64)
+ * - 1 #MCE exception frame (5 * uint64)
+ * - the largest stack use instantaneously possible by #MCE handling code
+ * - the largest stack use instantaneously possible by #NMI handling code
+ * - the largest stack use instantaneously possible by #DB handling code
+ * - one high-water uint32 used to detect stack overflows when debugging
+ *
+ * 184 bytes is slightly more than enough as of 2015/03/17 -- fjacobs.
+ */
+#define TINY_STACK_SIZE      184
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VMCrossPageData --
+ *
+ *      Data which is stored on the VMCrossPage.
+ *
+ *----------------------------------------------------------------------
+ */
+typedef
+#include "vmware_pack_begin.h"
+struct VMCrossPageData {
+   uint32   version;           // CROSSPAGE_VERSION
+   uint32   vmmonVersion;      // VMMON_VERSION
+
+   /*
+    * The following stacks and contexts are ordered for performance and code
+    * simplicity.  Both HostToVmm and VmmToHost strictly require this ordering.
+    * For any change to this data, both functions must be updated.
+    * Use VMX86_UCCOST to measure performance when changing the layout.
+    */
+
+   /* A tiny stack and the host context. */
+   uint64   hostTinyStack[TINY_STACK_SIZE / sizeof(uint64)];
+   uint64   hostCR3; /* Edge of context saved/restored in assembly. */
+   uint64   hostRBX;
+   uint64   hostRBP;
+   uint64   hostRSI;
+   uint64   hostRDI;
+   uint64   hostR12;
+   uint64   hostR13;
+   uint64   hostR14;
+   uint64   hostR15;
+   uint64   hostRSP;
+   uint16   hostSS;  /* Edge of context saved/restored in assembly. */
+   uint16   hostDS;  /* Not saved/restored in assembly switch */
+   uint16   hostES;  /* Not saved/restored in assembly switch */
+   uint16   hostPad;
+
+   /* A tiny stack and the monitor context. */
+   uint64   monTinyStack[TINY_STACK_SIZE / sizeof(uint64)];
+   uint64   monCR3; /* Edge of context saved/restored in assembly. */
+   uint64   monRBX;
+   uint64   monRBP;
+   uint64   monR12;
+   uint64   monR13;
+   uint64   monR14;
+   uint64   monR15;
+   uint64   monRSP;
+   uint16   monSS;  /* Edge of context saved/restored in assembly. */
+   uint16   monDS;  /* Not saved/restored in assembly switch */
+   uint16   monES;  /* Not saved/restored in assembly switch */
+   uint16   monPad;
+
+   uint64   crosspageMA;
+
+   uint64   hostDR[8];
+   LA64     crossPageLA;       // where host/PTP map the cross page
+   LA64     crossGDTLA;        // where host/PTP map the cross GDT
+   uint16   hostInitial64CS;
+   uint8    hostDRSaved;       // Host DR spilled to hostDR[x].
+   uint8    hostDRInHW;        // 0 -> shadowDR in h/w, 1 -> hostDR in h/w.
+   uint32   _pad0;
+
+   uint64   wsCR0;
+   uint64   wsCR4;
+
+   DTR64    crossGDTHKLADesc;   // always uses host kernel linear address
+   uint16   _pad1[3];
+   DTR64    monGDTR;
+   uint16   _pad2[3];
+   /* A hardcoded value for monitor %rip which facilitates backtraces. */
+   uint64   monRIP;
+   Task64   monTask;          /* vmm's task */
+
+   VMMPageTablePatch vmmPTP[MAX_SWITCH_PT_PATCHES]; /* page table patch */
+
+   /*
+    * The monitor may requests up to two actions when returning to the
+    * host.  The moduleCallType field and args encode a request for
+    * some action in the driver.  The vcpuSet field is an additional
+    * argument used in some calls.  The userCallType field (together
+    * with the RPC block) encodes a user call request.  The two
+    * requests are independent.  The user call is executed first, with
+    * the exception of MODULECALL_INTR which has a special effect.
+    */
+   ModuleCallType moduleCallType;
+   uint32         retval;
+   uint64         args[MODULECALL_NUM_ARGS];
+#if !defined(VMX86_SERVER)
+   VCPUSet        vcpuSet;
+#endif
+   int            userCallType;
+   uint32         pcpuNum;   /* Used as extra module call arg within vmmon. */
+
+#if !defined(VMX86_SERVER)
+   uint64 ucTimeStamps[UCCOST_MAX];
+#endif
+
+   SwitchedMSRState switchedMSRState;
+
+   /*
+    * The values in the shadow debug registers must match those in the
+    * hardware debug register immediately after a task switch in
+    * either direction.  They are used to minimize moves to and from
+    * the debug registers.
+    */
+   SharedUReg64     shadowDR[8];
+   uint8            shadowDRInHW; // bit n set iff %DRn == shadowDR[n]
+
+   /* TRUE if no bits are set in yieldVCPUs, FALSE otherwise. */
+   Bool             yieldVCPUsIsEmpty;
+   uint8            _pad3[6];
+
+   VCPUSet          yieldVCPUs;
+
+   /*
+    * Adjustment for machines where the hardware TSC does not run
+    * constantly (laptops) or is out of sync between different PCPUs.
+    * Updated as needed by vmmon.  See VMK_SharedData for the ESX
+    * analog, which is updated by the vmkernel.
+    */
+   RateConv_ParamsVolatile pseudoTSCConv;
+   VmAbsoluteTS            worldSwitchPTSC; // PTSC value immediately before
+                                            // last worldswitch.
+
+   VmAbsoluteTS timerIntrTS;    // PTSC of timer interrupt while in the vmm
+   VmAbsoluteTS hstTimerExpiry; // PTSC of host timer interrupt
+   VmAbsoluteTS monTimerExpiry; // PTSC of next MonTimer callback
+
+   Bool     retryWorldSwitch;   // TRUE -> return to host on host->vmm switch
+   /*
+    * TRUE if moduleCall was interrupted by signal. Only
+    * vmmon uses this field to remember that it should
+    * restart RunVM call, nobody else should look at it.
+    */
+   Bool     moduleCallInterrupted;
+   uint8    _pad4[6];
+
+   DTR64    switchHostIDTR;   // baseLA = switchHostIDT's host kernel LA
+   uint16   _pad5[3];
+   DTR64    switchMonIDTR;    // baseLA = switchMonIDT's monitor LA
+   uint16   _pad6[3];
+
+   /*
+    * Descriptors and interrupt tables for switchNMI handlers.  Each
+    * IDT has only enough space for the hardware exceptions; they are
+    * sized to accommodate 64-bit descriptors.
+    */
+   Gate64 switchHostIDT[NUM_EXCEPTIONS];      // hostCS:hostVA
+   Gate64 switchMonIDT[NUM_EXCEPTIONS];       // monCS:monVA
+   volatile Bool wsException[NUM_EXCEPTIONS]; // Tracks faults in worldswitch.
+   uint8         _pad7[4];
+   uint64        wsUD2;                       // IP of ud2 instr or 0 if unset.
+   uint64        specCtrl; /* host MSR_SPEC_CTRL value before world switch. */
+}
+#include "vmware_pack_end.h"
+VMCrossPageData;
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VMCrossPageCode --
+ *
+ *      Code which is stored on the VMCrossPage.
+ *
+ *----------------------------------------------------------------------
+ */
+
+#define CODE_BLOCK_SIZE (PAGE_SIZE - sizeof(VMCrossPageData) - \
+                         (2 + NUM_EXCEPTIONS) * CP_STUB_SIZE)
+
+typedef
+#include "vmware_pack_begin.h"
+struct VMCrossPageCode {
+   uint8         toVmmFunc[CP_STUB_SIZE];    // Fixed-position stubs to jump
+   uint8         toHostFunc[CP_STUB_SIZE];   // to world switch functions
+   uint8         gateStubs[NUM_EXCEPTIONS *
+                           CP_STUB_SIZE];    // Stubs for crossIDT.
+   uint8         codeBlock[CODE_BLOCK_SIZE]; // Code for worldswitch and
+                                             // fault handling.
+}
+#include "vmware_pack_end.h"
+VMCrossPageCode;
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VMCrossPage --
+ *
+ *      Data structure shared between the monitor and the module
+ *      that is used for crossing between the two.
+ *      Accessible as vm->crosspage (kernel module) and CROSS_PAGE
+ *      (monitor)
+ *
+ *      Must be exactly one page long.
+ *
+ *----------------------------------------------------------------------
+ */
+
+typedef
+#include "vmware_pack_begin.h"
+struct VMCrossPage {
+   VMCrossPageData crosspageData;
+   VMCrossPageCode crosspageCode;
+}
+#include "vmware_pack_end.h"
+VMCrossPage;
+
+#define CROSSPAGE_VERSION_BASE 0xc0e /* increment by 1 */
+#define CROSSPAGE_VERSION    ((CROSSPAGE_VERSION_BASE << 1) + WS_INTR_STRESS)
+
+#if !defined(VMX86_SERVER) && defined(VMM)
+#define CROSS_PAGE             ((VMCrossPage * const)VPN_2_VA(CROSS_PAGE_START))
+#define VMM_SWITCH_SHARED_DATA ((VMCrossPageData *)&CROSS_PAGE->crosspageData)
+#endif
+
+#define MX_WAITINTERRUPTED     3
+#define MX_WAITTIMEDOUT        2
+#define MX_WAITNORMAL          1  // Must equal one; see linux module code.
+#define MX_WAITERROR           0  // Use MX_ISWAITERROR() to test for error.
+
+// Any zero or negative value denotes error.
+#define MX_ISWAITERROR(e)      ((e) <= MX_WAITERROR)
+#endif

vmmon-only/include/modulecallstructs.h

@@ -0,0 +1,98 @@
+/*********************************************************
+ * Copyright (C) 2006,2009-2011,2013-2015,2020 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * modulecallstructs.h --
+ *
+ *
+ *      Data structures that need to be included in modulecall.h
+ *      as well as the vmkernel.
+ *
+ */
+
+#ifndef _MODULECALLSTRUCTS_H_
+#define _MODULECALLSTRUCTS_H_
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMCORE
+
+#include "includeCheck.h"
+
+#include "vm_basic_types.h"
+
+/*
+ *      Flags indicating switched MSR status.
+ *
+ * UNUSED    - Not used by the monitor (yet). [This is a value, not a flag.]
+ * USED      - Hardware MSR is used by the monitor.
+ * RESTORED  - Monitor value is restored on world switch into the VMM.
+ * SHADOWED  - Live monitor value is always shadowed in the SwitchedMSRState.
+ *
+ */
+
+#define SWITCHED_MSR_FLAG_UNUSED           0
+#define SWITCHED_MSR_FLAG_USED             1
+#define SWITCHED_MSR_FLAG_RESTORED         2
+#define SWITCHED_MSR_FLAG_SHADOWED         4
+
+/*
+ * Note: If you add an msr to this list, please also ensure that
+ *       hardware support for the msr is properly indicated in
+ *       both the monitor (MonMSRIsSupported) and in the vmkernel
+ *       (world switch msrSupported array).
+ */
+#define SWITCHED_MSRS       \
+   SWMSR(MSR_SYSENTER_CS)   \
+   SWMSR(MSR_SYSENTER_EIP)  \
+   SWMSR(MSR_SYSENTER_ESP)  \
+   SWMSR(MSR_STAR)          \
+   SWMSR(MSR_LSTAR)         \
+   SWMSR(MSR_CSTAR)         \
+   SWMSR(MSR_SFMASK)        \
+   SWMSR(MSR_TSC_AUX)       \
+   SWMSR(MSR_BD_TSC_RATIO)
+
+/*
+ *      Data structures for dealing with the context-switched MSRs that need
+ *      to be specially handled.  While the MSR definitions themselves
+ *      are part of the x86 architecture, our handling of them (and hence
+ *      these data structures) is an implementation detail.
+ */
+
+
+typedef enum SwitchedMSR {
+#define SWMSR(msr) SWITCHED_##msr,
+   SWITCHED_MSRS
+#undef SWMSR
+   NUM_SWITCHED_MSRS
+} SwitchedMSR;
+
+/*
+ * Switched MSR values for each [vp]CPU.
+ */
+typedef struct SwitchedMSRValues {
+   uint64 a[NUM_SWITCHED_MSRS];
+} SwitchedMSRValues;
+
+typedef struct SwitchedMSRState {
+   SwitchedMSRValues smv;
+   uint8             flags[NUM_SWITCHED_MSRS];
+} SwitchedMSRState;
+
+#endif

vmmon-only/include/monAddrLayout.h

@@ -0,0 +1,75 @@
+/*********************************************************
+ * Copyright (C) 2005-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * monAddrLayout.h --
+ *
+ *    Address layout of the monitor.
+ */
+
+#ifndef _MON_ADDR_LAYOUT_H
+#define _MON_ADDR_LAYOUT_H
+
+#include "vm_pagetable.h"
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_USERLEVEL
+#include "includeCheck.h"
+
+#pragma pack(push, 1)
+typedef struct VMM64_AddrLayout {
+   /*
+    *All address are VPNs and all lengths are numPages
+    */
+   uint64 monBase;            // MONITOR_BASE_VPN
+   uint64 monL5Start;         // MON_PAGE_TABLE_L5_START
+   uint32 monL5Len;           // MON_PAGE_TABLE_L5_LEN
+   uint64 monL4Start;         // MON_PAGE_TABLE_L4_START
+   uint32 monL4Len;           // MON_PAGE_TABLE_L4_LEN
+   uint64 monL3Start;         // MON_PAGE_TABLE_L3_START
+   uint32 monL3Len;           // MON_PAGE_TABLE_L3_LEN
+   uint64 monL2Start;         // MON_PAGE_TABLE_L2_START
+   uint32 monL2Len;           // MON_PAGE_TABLE_L2_LEN
+   uint64 monL1Start;         // MON_PAGE_TABLE_L1_START
+   uint32 monL1Len;           // MON_PAGE_TABLE_L1_LEN
+#ifndef VMX86_SERVER
+   uint64 monCpStart;         // CROSS_PAGE_START
+#endif
+} VMM64_AddrLayout;
+#pragma pack(pop)
+
+#define VMM_MONAS_4LP_FIRST_L4OFF   1
+#define VMM_MONAS_4LP_LAST_L4OFF  130
+
+#define VMM_MONAS_5LP_FIRST_L5OFF   1
+#define VMM_MONAS_5LP_LAST_L5OFF   34
+
+static INLINE PT_Level
+MonAS_GetPagingLevel(void)
+{
+   return PT_LEVEL_4;
+}
+
+static INLINE Bool
+MonAS_Uses5LevelPaging(void)
+{
+   return MonAS_GetPagingLevel() == PT_LEVEL_5;
+}
+#endif

vmmon-only/include/monLoader.h

@@ -0,0 +1,340 @@
+/*********************************************************
+ * Copyright (C) 2015-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * monLoader.h --
+ *
+ *      Describes the monitor loader, its header and support code for
+ *      processing this header.
+ *
+ *
+ * Overview
+ * ========
+ *
+ * Before the monitor or its in-context bootstrap code can run, a monitor
+ * address space must be created and partially populated.  This address space
+ * contains code and data statically built, empty but allocated space, shared
+ * and run-time initialized content.  The monitor loader header regularizes
+ * encoding of address space information, allowing a common representation and
+ * common code to be re-used for different contexts (vmmon for hosted
+ * and vmkernel for ESX).
+ *
+ * The Header
+ * ==========
+ *
+ * The monitor loader header contains a sequence of entries describing content.
+ * Each entry has a start and end VPN, an optional content source, permission
+ * flags to be applied when mapping, an optional subIndex specifying a shared
+ * region (for shared content types) and a flag specifying whether to process
+ * this entry for all VCPUs or just the bootstrap processor.
+ *
+ * Content types
+ * =============
+ *
+ * ADDRSPACE: Must be first.  Describes the address space in which all other
+ * entries reside.  Page tables from L4 to L1 will be allocated, in order,
+ * eagerly.  These page tables will be wired with the permissions of this entry
+ * from L4 to L1.  This allows easy access to the L1E for any VPN in the space.
+ *
+ * ML_CONTENT_PAGETABLE_Lx: Maps the page tables for a given level.  It must be
+ * exactly the size of the tables preallocated by ADDRSPACE for the level.
+ *
+ * ML_CONTENT_ALLOCZERO: Allocates new MPNs, zeroes the pages and maps them.
+ *
+ * ML_CONTENT_COPY: Allocates new MPNs, maps them and copies from the specified
+ * source.
+ *
+ * ML_CONTENT_SHARE: Memory provided by user, host kernel, or VMM blob,
+ * and mapped into the monitor address space.  The subIndex field
+ * specifies which region, as multiple regions may be shared for each
+ * source.
+ *
+ * Processing
+ * ==========
+ *
+ * The monitor loader can be built in different contexts.  Each context must
+ * provide implementations of callout functions.  To fully build the context,
+ * callouts for memory allocation, access and reporting of resources will be
+ * added, as will accounting functionality.
+ *
+ * Callouts
+ * ========
+ * 
+ * Memory-accessing callouts take a Vcpuid as memory is tracked per-VCPU.
+ *
+ * MonLoaderCallout_AllocMPN(MPN): Allocate a new MPN.
+ * MonLoaderCallout_CleanUp(): Release temporary MonLoader callout resources.
+ * MonLoaderCallout_CopyFromBlob(blobOffset, size, MPN): Copy blob contents.
+ * MonLoaderCallout_FillPage(pattern, mpn): fill a page with a pattern.
+ * MonLoaderCallout_GetPTE(MPN, index, *pte): Get indexth PTE in a PT MPN.
+ * MonLoaderCallout_GetPageRoot(vcpu): Get a VCPU's page root.
+ * MonLoaderCallout_ImportPage(MPN): Import an MPN for tracking and future use.
+ * MonLoaderCallout_Init(): Initialize MonLoader callouts.
+ * MonLoaderCallout_GetSharedHostPage(subIdx, page): Get shared host page MPN.
+ * MonLoaderCallout_GetSharedUserPage(subIdx, page): Get shared user page MPN.
+ *
+ * Building vs Importing
+ * =====================
+ *
+ * While legacy VMX is still responsible for building part of the monitor
+ * context, the monitor loader must cooperate.  Rather than building the
+ * context itself, the monitor loader verifies its environment matches its
+ * header's expectations.  This includes verifying sufficient allocations,
+ * permissions of page table wiring and page table self-mapping.
+ *
+ * Errors
+ * ======
+ *
+ * To avoid ASSERTs across contexts, errors are returned and clean-up is done
+ * carefully if processing fails at any stage.  Some errors are debug-only and
+ * essentially verify invariants for debug builds.  The line in the table at
+ * which the error was encountered is returned for logging and debugging.
+ *
+ * Compatibility
+ * =============
+ *
+ * bootstrap-offsets.pl relies on MonLoaderContentType, MonLoaderSourceType and
+ * the structure of MonLoaderEntry to determine and set blob offsets for
+ * ML_CONTENT_COPY + ML_SOURCE_BLOB.  If these are changed, the script should
+ * be updated accordingly.
+ */
+
+#ifndef _MON_LOADER
+#define _MON_LOADER
+
+#include "vm_basic_types.h"
+#include "vm_pagetable.h"
+#include "vcpuid.h"   /* Vcpuid */
+
+#if defined VM_X86_64
+#include "x86paging_64.h"
+#elif defined VM_ARM_64
+#include "arm64_vmsa.h"
+#endif
+
+#define ML_NAME_MAX 16
+
+/* ML perms are simple and abbreviated. */
+
+#if defined VM_X86_64
+
+#define ML_PERM_RWX (PTE_P | PTE_RW)
+#define ML_PERM_RW  (PTE_P | PTE_RW | PTE_NX)
+#define ML_PERM_RO  (PTE_P |          PTE_NX)
+#define ML_PERM_RX   PTE_P
+
+#define ML_PERM_TBL   ML_PERM_RWX
+#define ML_PERM_MASK (PTE_P| PTE_RW | PTE_NX | PTE_US)
+
+#define ML_PERM_PRESENT(_flags)     (((_flags) & PTE_P) != 0)
+#define ML_PERM_WRITEABLE(_flags)   (((_flags) & PTE_RW) != 0)
+
+#define ML_PTE_2_PFN(_pte)          LM_PTE_2_PFN(_pte)
+
+#elif defined VM_ARM_64
+
+#define _ML_PERM_COMMON (ARM_PTE_BLOCK_AP(ARM_AP_PL0)   | ARM_PTE_BLOCK_AF | \
+                         ARM_PTE_BLOCK_SH(ARM_SH_OUTER) | ARM_PTE_BLOCK_L3_TYPE)
+#define ARM_PTE_BLOCK_AP_RO ARM_PTE_BLOCK_AP(ARM_AP_RO)
+
+#define ML_PERM_RW   (_ML_PERM_COMMON |                       ARM_PTE_BLOCK_XN)
+#define ML_PERM_RO   (_ML_PERM_COMMON | ARM_PTE_BLOCK_AP_RO | ARM_PTE_BLOCK_XN)
+#define ML_PERM_RX   (_ML_PERM_COMMON | ARM_PTE_BLOCK_AP_RO                   )
+
+#define ML_PERM_TBL   ML_PERM_RW
+#define ML_PERM_MASK (_ML_PERM_COMMON | ARM_PTE_BLOCK_AP_RO | ARM_PTE_BLOCK_XN)
+
+#define ML_PERM_PRESENT(_flags)   (((_flags) & ARM_PTE_VALID) != 0)
+#define ML_PERM_WRITEABLE(_flags) (((_flags) & ARM_PTE_BLOCK_AP_RO) == 0)
+
+#define ML_PTE_2_PFN(_pte)    (((_pte) & ARM_PTE_PFN_MASK) >> PT_PTE_PFN_SHIFT)
+
+#endif
+
+#define ML_PERMS_MATCH(x,p) (((x) & ML_PERM_MASK) == ((p) & ML_PERM_MASK))
+
+#define LOADER_HEADER_MAGIC 0x8675309E98675309
+
+typedef enum {
+   ML_CONTENT_INVALID = 0,
+   ML_CONTENT_ADDRSPACE,     /* The static address space. */
+   ML_CONTENT_ALLOCZERO,     /* Allocate, zero-fill and map. */
+   ML_CONTENT_COPY,          /* Copy data from external source. */
+   ML_CONTENT_PAGETABLE_L4,  /* Mappings for level 4 page tables in AS. */
+   ML_CONTENT_PAGETABLE_L3,  /* Mappings for level 3 page tables in AS. */
+   ML_CONTENT_PAGETABLE_L2,  /* Mappings for level 2 page tables in AS. */
+   ML_CONTENT_PAGETABLE_L1,  /* Mappings for level 1 page tables in AS. */
+   ML_CONTENT_SHARE,         /* Share data from external source. */
+} MonLoaderContentType;
+
+#define CONTENT_TO_PTLEVEL(x) (x == ML_CONTENT_PAGETABLE_L4 ? PT_LEVEL_4 : \
+                               x == ML_CONTENT_PAGETABLE_L3 ? PT_LEVEL_3 : \
+                               x == ML_CONTENT_PAGETABLE_L2 ? PT_LEVEL_2 : \
+                               x == ML_CONTENT_PAGETABLE_L1 ? PT_LEVEL_1 : \
+                               0)
+
+/* Sources of content, for pages not zeroed, unmapped or otherwise special. */
+typedef enum MonLoaderSourceType {
+   ML_SOURCE_INVALID  = 0x0,
+   ML_SOURCE_NONE,
+   ML_SOURCE_BLOB,
+   ML_SOURCE_USER,
+   ML_SOURCE_HOST,
+} MonLoaderSourceType;
+
+/*
+ * NOTE: When modifying MonLoaderEntry, corresponding changes must be
+ * made to vmcore/make/misc/bootstrap-offsets.pl.
+ */
+typedef struct {
+   MonLoaderContentType content;           /* Content type. */
+   MonLoaderSourceType  source;            /* Source, if any, for content. */
+   VPN                  monVPN;            /* Destination in monitor AS. */
+   uint64               monPages;          /* Size in pages. */
+   uint64               flags;             /* PTE permissions */
+   unsigned             allocs;            /* MPNs allocated (all VCPUs). */
+   /* Set for source ML_SOURCE_BLOB: */
+   struct blobSrc {
+      uint64 offset; /* offset within the blob, in bytes. */
+      uint64 size;   /* size of content, in bytes. */
+   } blobSrc;
+   uint64               bspOnly;           /* Process only on BSP. */
+   uint64               subIndex;          /* Region ID for ML_CONTENT_COPY. */
+} MonLoaderEntry;
+
+
+/*
+ * Packed for easy consumption by bootstrap-offsets.pl. If the contents of the
+ * MonLoaderHeader struct are changed then $HEADER_SIZE must be updated
+ * accordingly in bootstrap-offsets.pl.
+ */
+#pragma pack(push, 1)
+typedef struct MonLoaderHeader {
+   uint64         magic;
+   uint32         entrySize;
+   uint32         count;
+   /* cs:rip */
+   uint16         codeSelector;
+   VA64           codeEntrypoint;
+   /* ss:rip */
+   uint16         stackSelector;
+   VA64           stackEntrypoint;
+   LPN64          monStartLPN;
+   LPN64          monEndLPN;
+   MonLoaderEntry entries[];
+} MonLoaderHeader;
+#pragma pack(pop)
+
+/* Environment context structure, defined by the environment. */
+struct MonLoaderEnvContext;
+
+/* Callout prototypes */
+MPN  MonLoaderCallout_AllocMPN(struct MonLoaderEnvContext *, Vcpuid);
+void MonLoaderCallout_CleanUp(struct MonLoaderEnvContext *);
+Bool MonLoaderCallout_CopyFromBlob(struct MonLoaderEnvContext *, uint64,
+                                   size_t, MPN, Vcpuid);
+Bool MonLoaderCallout_FillPage(struct MonLoaderEnvContext *, uint8, MPN,
+                               Vcpuid);
+MPN  MonLoaderCallout_GetPageRoot(struct MonLoaderEnvContext *, Vcpuid);
+Bool MonLoaderCallout_GetPTE(struct MonLoaderEnvContext *, MPN, unsigned,
+                             Vcpuid, PT_L1E *);
+Bool MonLoaderCallout_ImportPage(struct MonLoaderEnvContext *, MPN, Vcpuid);
+Bool MonLoaderCallout_Init(void *, struct MonLoaderEnvContext **, unsigned);
+Bool MonLoaderCallout_MapMPNInPTE(struct MonLoaderEnvContext *, MPN, unsigned,
+                                  uint64, MPN, Vcpuid);
+MPN  MonLoaderCallout_GetSharedUserPage(struct MonLoaderEnvContext *, uint64,
+                                        unsigned, Vcpuid);
+MPN  MonLoaderCallout_GetSharedHostPage(struct MonLoaderEnvContext *, uint64,
+                                        unsigned, Vcpuid);
+MPN  MonLoaderCallout_GetBlobMpn(struct MonLoaderEnvContext *, uint64);
+
+typedef enum MonLoaderError {
+   ML_OK = 0,
+   ML_ERROR_ADDRSPACE_TOO_LARGE,
+   ML_ERROR_ALLOC,
+   ML_ERROR_ALREADY_MAPPED,
+   ML_ERROR_ARGS,
+   ML_ERROR_CALLOUT_INIT,
+   ML_ERROR_CALLOUT_COPY,
+   ML_ERROR_CALLOUT_GETPTE,
+   ML_ERROR_CALLOUT_MAPINPTE,
+   ML_ERROR_CALLOUT_PAGEROOT_GET,
+   ML_ERROR_CALLOUT_ZERO,
+   ML_ERROR_CONTENT_INVALID,
+   ML_ERROR_CONTEXT_INIT,
+   ML_ERROR_DUPLICATE,
+   ML_ERROR_INVALID_VPN,
+   ML_ERROR_MAGIC,
+   ML_ERROR_MAP,
+   ML_ERROR_NO_ADDRSPACE,
+   ML_ERROR_PAGE_TABLE_IMPORT,
+   ML_ERROR_PAGE_TABLE_MAP_SIZE,
+   ML_ERROR_PAGE_TABLE_VERIFY,
+   ML_ERROR_SHARE,
+   ML_ERROR_SIZE,
+   ML_ERROR_SOURCE_INVALID,
+   ML_ERROR_TABLE_MISSING,
+} MonLoaderError;
+
+
+/*
+ * Values above shared area subindices for sharing of MonLoaderHeader
+ * and htSchedStateMap.
+ */
+#define MONLOADER_HEADER_IDX 6
+#define MONLOADER_HT_MAP_IDX 7
+
+
+MonLoaderError MonLoader_Process(MonLoaderHeader *header, unsigned numVCPUs,
+                                 void *args, unsigned *line, Vcpuid *vcpu);
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoader_GetFixedHeaderSize --
+ *
+ *      Returns the size of the fixed portion of MonLoaderHeader.
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE size_t
+MonLoader_GetFixedHeaderSize(void)
+{
+   return sizeof(MonLoaderHeader);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MonLoader_GetFullHeaderSize --
+ *
+ *      Returns the size of the full MonLoaderHeader, including the
+ *      variable size portion.
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE size_t
+MonLoader_GetFullHeaderSize(MonLoaderHeader *header)
+{
+   return MonLoader_GetFixedHeaderSize() +
+          header->count * sizeof(MonLoaderEntry);
+}
+
+#endif /* !_MON_LOADER */

vmmon-only/include/monLoaderLog.h

@@ -0,0 +1,50 @@
+/*********************************************************
+ * Copyright (C) 2016-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * monLoaderLog.h --
+ *
+ *      Logging definitions for MonLoader.
+ */
+
+#ifndef _MONLOADER_LOG_H_
+#define _MONLOADER_LOG_H_
+
+#if defined VMKERNEL
+
+#define LOGLEVEL_MODULE MonLoader
+#include "log.h"
+
+#elif defined VMMON
+
+#include "vm_assert.h"
+#define ML_LOGLEVEL_VMMON 0 /* MonLoader/vmmon loglevel. */
+
+#undef LOG
+#define LOG(_min, _fmt, ...)                         \
+do {                                                 \
+   if (vmx86_log && ML_LOGLEVEL_VMMON >= (_min)) {   \
+      Log(_fmt "\n", ## __VA_ARGS__);                \
+   }                                                 \
+} while (0)
+
+#else /* !defined VMKERNEL && !defined VMMON */
+#error MonLoader cannot be built as part of this environment
+#endif
+
+#endif /* _MONLOADER_LOG_H_ */

vmmon-only/include/mon_assert.h

@@ -0,0 +1,203 @@
+/*********************************************************
+ * Copyright (C) 2007-2015,2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef _MON_ASSERT_H_
+#define _MON_ASSERT_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#include "vm_assert.h"
+#include "vm_basic_asm.h"
+
+/*
+ * Monitor Source Location
+ *
+ * The monitor encodes source locations -- file name & line number --
+ * in just 32 bits; the process is arcane enough that it deserves a
+ * little discussion.
+ *
+ *   o The ASSERT family of macros are expanded in the monitor to take
+ *     an 'Assert_MonSrcLoc' rather than the standard '<file>,
+ *     <line-number>' couplet.
+ *
+ *   o The '<file>, <line-number>' couplet is encoded into
+ *     Assert_MonSrcLoc, which is an unsigned 32-bit integer.
+ *
+ *   o The upper 16-bits of Assert_MonSrcLoc are the line number.
+ *
+ *     Source lines above 65535 will be silently masked to 16-bits.
+ *
+ *   o The lower 16-bits of Assert_MonSrcLoc are the offset to the file
+ *     name from the start of the file name table.
+ *
+ *     This, of course, implies that the size of the table containing
+ *     the file names cannot exceed 64K.
+ *
+ *   o If we use '__FILE__' directly, gcc will coalesce all equivalent
+ *     strings into a single occurrence (in '.rodata').
+ *
+ *     Using the full pathname for __FILE__ is undesirable because
+ *     different source trees frequently have different path name
+ *     lengths, and this causes the 'monitor-modular-size' script to
+ *     report differences in '.rodata'.
+ *
+ *   o Each __FILE__ is put into its own .assert_pathname_##__FILE
+ *     section.  The monitor's runtime linker (not ld) then uses the name
+ *     of this section to recover the name of the corresponding source file.
+ *
+ *   o At run time, prior to loading, when our linker is creating an
+ *     executable image of the monitor and extensions, all the file
+ *     names are extracted from these sections, the '${VMTREE}' prefix
+ *     is removed, and the resulting table of shortened file names is
+ *     added to a non-loadable section called '.assert_pathnames' in the
+ *     VMX linker (not the bootstrap linker which does not need them).
+ *
+ *     Further, during linkage, each relocation to the original
+ *     section containing the path name is modified so that the low
+ *     16-bits contain an offset from the start of the .assert_pathnames
+ *     section rather than the base of the original containing section.
+ *
+ *     Only three types of relocations to the assertion strings are
+ *     supported (32-bit PC-relative and 32-bit/64-bit absolute) because that
+ *     is all the compiler has been seen to generate.
+ */
+
+#define ALL_ASSERT_TYPES \
+   ADEF(AssertType_AssertPanic,          _AssertPanicFmt),                 \
+   ADEF(AssertType_AssertAssert,         _AssertAssertFmt),                \
+   ADEF(AssertType_AssertVerify,         _AssertVerifyFmt),                \
+   ADEF(AssertType_AssertNotImplemented, _AssertNotImplementedFmt),        \
+   ADEF(AssertType_AssertNotReached,     _AssertNotReachedFmt),            \
+   ADEF(AssertType_AssertPanicBug,       _AssertPanicFmt " bugNr=%d"),     \
+   ADEF(AssertType_AssertAssertBug,      _AssertAssertFmt " bugNr=%d"),    \
+   ADEF(AssertType_AssertVerifyBug,      _AssertVerifyFmt " bugNr=%d"),    \
+   ADEF(AssertType_AssertNotImplementedBug,                                \
+        _AssertNotImplementedFmt " bugNr=%d"),                             \
+   ADEF(AssertType_AssertNotReachedBug,  _AssertNotReachedFmt " bugNr=%d"),\
+   ADEF(AssertType_AssertNotTested,      _AssertNotTestedFmt)
+
+typedef uint32 Assert_MonSrcLoc;
+
+#define ADEF(type, fmt) type
+typedef enum Assert_Type {  
+   ALL_ASSERT_TYPES
+} Assert_Type;
+#undef ADEF
+
+typedef struct Assert_Info {
+   VA faultAddr;
+   struct {
+      Assert_Type type:4;
+      int bugNr:28;
+   } misc;
+   Assert_MonSrcLoc loc;
+} Assert_Info;
+
+#define ASSERT_MONSRCFILEOFFSET(loc)    LOWORD(loc)
+#define ASSERT_MONSRCLINE(loc)          HIWORD(loc)
+
+#define ASSERT_NULL_MONSRCLOC     0             // there is never line 0
+
+#if defined(VMM) && !defined(MONITOR_APP) // {
+/*
+ * Assertion information is collected in a non-loadable section
+ * named .assert_info.  Each record in this section contains
+ * a VMM address, an assertion type, an optional bug number, and
+ * the MonSrcLoc described previously.  The VMM address is a key
+ * used by the VMX to look up the information associated with
+ * a particular assertion failure.
+ *
+ * Assertion failures are fired by executing a ud2 instruction.
+ *
+ * For assertions which always result in a terminal user RPC, we use
+ * __builtin_trap to generate the ud2, so that gcc knows that the
+ * subsequent code is unreachable.  For assertions which are
+ * recoverable (e.g any assertion triggered on the BackToHost path),
+ * we generate the ud2 manually, so that gcc will treat the subsequent
+ * code as reachable.
+ *
+ * The memory barriers work around a gcc bug that results from having
+ * to continue past an assertion.  Without these barriers, gcc has been
+ * seen to hoist code into the failing arm of the assertion, where it
+ * can then tell that, because of the assertion failure, the code ends
+ * up accessing an array out of bounds.
+ *
+ * Assertion failures for the monitor's bootstrap are reduced to panics
+ * logging the current %rip.  As such, no .assert_info section is created
+ * and VMX merely processes the panic.
+ */
+
+#ifndef VMM_BOOTSTRAP
+/*
+ * The portion of the __attribute__ line after __FILE__ is there so that
+ * the .assert_pathname_* sections are not marked as ALLOC, since we only
+ * need them in the vmx and do not need them loaded.
+ */
+#define __VMM__FILE__SECTION \
+      __attribute__((section (".assert_pathname_" __FILE__ ",\"\"#")))
+#define __VMM__FILE__ ({                                                \
+         static __VMM__FILE__SECTION const char file[] = "";            \
+         file;                                                          \
+      })
+
+#define ASSERT_RECORDINFO(assembly, assertType, bugNr)                   \
+   __asm__ __volatile__(".pushsection .assert_info;"                     \
+                        ".quad 0f;"                                      \
+                        ".long %c[type] + (%c[bug] << 4);"               \
+                        ".long (%c[line] << 16) + %c[file];"             \
+                        ".popsection;"                                   \
+                        "0: " assembly : :                               \
+                        [line] "i" (__LINE__),                           \
+                        [file] "i" (__VMM__FILE__),                      \
+                        [type] "i" (assertType),                         \
+                        [bug]  "i" (bugNr))
+#else
+
+extern uint64 bsAssertRIP;
+#define ASSERT_RECORDINFO(assembly, assertType, bugNr)                   \
+   __asm__ __volatile__("lea 0(%%rip), %0\n\t"                           \
+                        : "=r"(bsAssertRIP));                            \
+   Panic("Bootstrap: %s failure at rip=0x%lx",                           \
+         assertType == AssertType_AssertVerify ? "VERIFY" : "ASSERT",    \
+         bsAssertRIP);
+#endif /* VMM_BOOTSTRAP */
+
+#define _ASSERT_PANIC(name)                                              \
+   ({COMPILER_MEM_BARRIER();                                             \
+     ASSERT_RECORDINFO("ud2", AssertType_##name, 0);})
+
+#define _ASSERT_PANIC_NORETURN(name)                                     \
+   ({COMPILER_MEM_BARRIER();                                             \
+     ASSERT_RECORDINFO("", AssertType_##name, 0);                        \
+     __builtin_trap();})
+
+#define _ASSERT_PANIC_BUG(bug, name)                                     \
+   ({COMPILER_MEM_BARRIER();                                             \
+     ASSERT_RECORDINFO("ud2", AssertType_##name##Bug, bug);})
+
+#define _ASSERT_PANIC_BUG_NORETURN(bug, name)                            \
+   ({COMPILER_MEM_BARRIER();                                             \
+     ASSERT_RECORDINFO("", AssertType_##name##Bug, bug);                 \
+     __builtin_trap();})
+#endif // VMM && !MONITOR_APP}
+
+#endif // _MON_ASSERT_H_

vmmon-only/include/monitorAction_exported.h

@@ -0,0 +1,43 @@
+/*********************************************************
+ * Copyright (C) 2010-2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef _MONITORACTION_EXPORTED_H_
+#define _MONITORACTION_EXPORTED_H_
+
+#define INCLUDE_ALLOW_VMX
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#include "includeCheck.h"
+
+#include "vm_basic_types.h"
+
+#if defined __cplusplus
+extern "C" {
+#endif
+
+#define MONACTION_INVALID  MAX_UINT32
+
+typedef uint32 MonitorIdemAction;
+
+#if defined __cplusplus
+} // extern "C"
+#endif
+
+#endif // _MONITORACTION_EXPORTED_H_

vmmon-only/include/numa_defs.h

@@ -0,0 +1,83 @@
+/*********************************************************
+ * Copyright (C) 2006,2009-2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * numa_defs.h --
+ *	This is the internal header file for the NUMA module.
+ */
+
+#ifndef _NUMA_DEFS_H
+#define _NUMA_DEFS_H
+
+#define INCLUDE_ALLOW_VMX
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+
+#include "includeCheck.h"
+#include "vm_basic_types.h"
+#include "vm_basic_defs.h"
+#include "cpu_defs.h"
+
+/* Machine NUMA nodes */
+typedef uint32 NUMA_Node;
+typedef uint32 NUMA_PxmID;
+typedef uint64 NUMA_NodeMask;
+typedef uint8  NUMA_MemRangeID;
+
+/*
+ * Constants
+ */
+#define NUMA_MAX_NODES              64
+#define NUMA_MAX_CPUS_PER_NODE      (vmx86_server ? MAX_PCPUS : 32)
+#ifdef VMKBOOT
+#define NUMA_MAX_MEM_RANGES         64
+#else
+#define NUMA_MAX_MEM_RANGES         256
+#endif
+#define INVALID_NUMANODE            ((NUMA_Node)-1)
+#define NUMA_NODE_MASK_ANY          ((NUMA_NodeMask)-1)
+#define NUMA_NODE_MASK_NONE         ((NUMA_NodeMask)0)
+
+
+/*
+ * Structures
+ */
+typedef struct {
+   MPN          startMPN;
+   MPN          endMPN;
+   NUMA_PxmID   id;
+   Bool         isReliable;
+   Bool         isVolatile;
+} NUMA_MemRange;
+
+typedef struct NUMA_MemRangesList {
+   uint64        numMemRanges;
+   NUMA_MemRange memRange[NUMA_MAX_MEM_RANGES];
+} NUMA_MemRangesList;
+
+
+typedef struct NUMA_MemRangesListRef {
+   uint64        numMemRanges;
+   const NUMA_MemRange *memRange;
+} NUMA_MemRangesListRef;
+
+#endif // _NUMA_DEFS_H

vmmon-only/include/overheadmem_types.h

@@ -0,0 +1,131 @@
+/*********************************************************
+ * Copyright (C) 2001-2013 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * overheadmem_types.h
+ *
+ *	Types for tracking memory overheads.
+ */
+
+#ifndef _OVERHEADMEM_TYPES_H
+#define _OVERHEADMEM_TYPES_H
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_MODULE
+#include "includeCheck.h"
+
+#include "vm_basic_types.h"
+
+/*
+ * There are 4 types of memory we lock on the host.  Memory can be Mem_Mapped in
+ * the vmx, anonymous memory for use by monitor is not mapped permanently in any
+ * address space, guest memory regions other than main memory (can be
+ * locked/unlocked on hosted but not on ESX), and main memory which can be
+ * locked/unlocked in hosted and esx.
+ *
+ * In addition, the vmx may malloc memory or declare (large) static structures.
+ * Neither of these is locked on hosted platforms and the hostOS may swap it.
+ * Therefore, on hosted platforms we do not track this memory and instead
+ * include a working set component (sched.mem.hosted.perVMOverheadMBs).
+ * On ESX, this memory must be accounted for so we account it to user
+ * (nonpaged) overhead.  At present, the accounting is extremely coarse,
+ * and only aggregate sizes are hard-coded (see PR363997).
+ */
+typedef enum OvhdMemType {
+   OvhdMem_memmap,
+   OvhdMem_anon,
+   OvhdMem_guest,
+   OvhdMem_mainmem,
+   OvhdMem_malloc,
+   OvhdMem_static,
+   OvhdMem_text,
+   NumOvhdMemTypes
+} OvhdMemType;
+
+#define OvhdMemMask(type)    (1 << type)
+
+#define OVHDMEM_NONE      0x0
+#define OVHDMEM_MEMMAP    0x1 // OvhdMemMask(OvhdMem_memmap)
+#define OVHDMEM_ANON      0x2 // OvhdMemMask(OvhdMem_anon)
+#define OVHDMEM_GUEST     0x4 // OvhdMemMask(OvhdMem_guest)
+#define OVHDMEM_MAINMEM   0x8 // OvhdMemMask(OvhdMem_mainmem)
+#define OVHDMEM_MALLOC   0x10 // OvhdMemMask(OvhdMem_malloc)
+#define OVHDMEM_STATIC   0x20 // OvhdMemMask(OvhdMem_static)
+#define OVHDMEM_TEXT     0x40 // OvhdMemMask(OvhdMem_text)
+#define OVHDMEM_ALL_USER (OVHDMEM_MEMMAP | OVHDMEM_GUEST | OVHDMEM_MAINMEM | \
+                          OVHDMEM_MALLOC | OVHDMEM_STATIC | OVHDMEM_TEXT)
+#define OVHDMEM_ALL      (OVHDMEM_ALL_USER | OVHDMEM_ANON)
+
+/* ... and four categories of memory sources. */
+typedef enum OvhdMemCategory {
+   OvhdMemCat_paged,
+   OvhdMemCat_nonpaged,
+   OvhdMemCat_excluded,
+   OvhdMemCat_anonymous,
+   NumOvhdMemCategories
+} OvhdMemCategory;
+
+#define OVHDMEM_PAGED     (OVHDMEM_MALLOC  | OVHDMEM_STATIC)
+#define OVHDMEM_NONPAGED  (OVHDMEM_GUEST   | OVHDMEM_MEMMAP)
+#define OVHDMEM_EXCLUDED  (OVHDMEM_MAINMEM | OVHDMEM_TEXT)
+
+#if ((OVHDMEM_PAGED & OVHDMEM_NONPAGED) != 0)    ||                          \
+    ((OVHDMEM_NONPAGED & OVHDMEM_EXCLUDED) != 0) ||                          \
+    ((OVHDMEM_PAGED & OVHDMEM_EXCLUDED) != 0)    ||                          \
+    ((OVHDMEM_PAGED | OVHDMEM_NONPAGED | OVHDMEM_EXCLUDED | OVHDMEM_ANON) != \
+              OVHDMEM_ALL)
+#error Overheadmem categories do not form a partition of the overheads
+#endif
+
+/* Categories of overhead for 32-bit and 64-bit mode. */
+typedef struct OvhdMem_Overheads {
+   uint32 paged;
+   uint32 nonpaged;
+   uint32 anonymous;
+   uint32 text;
+} OvhdMem_Overheads;
+
+typedef struct OvhdMem_Deltas {
+   int32 paged;
+   int32 nonpaged;
+   int32 anonymous;
+} OvhdMem_Deltas;
+
+
+/* Types for tracking vmx (user) overheads. */
+
+#define OVHDMEM_MAX_NAME_LEN 36
+
+/* Types for tracking vmm overheads. */
+
+typedef struct OvhdMemUsage {
+   uint32 reserved; // pages
+   uint32 used;     // pages
+} OvhdMemUsage;
+
+typedef struct OvhdMemNode {
+   OvhdMemUsage usage;               // allocated and rsvd bytes for source
+   OvhdMemUsage maxUsage;            // max allocated and rsvd bytes for source
+   char name[OVHDMEM_MAX_NAME_LEN];  // name of overhead source
+   OvhdMemType type;                 // how/where memory for source is managed
+} OvhdMemNode;
+
+#endif

vmmon-only/include/pageLock_defs.h

@@ -0,0 +1,105 @@
+/*********************************************************
+ * Copyright (C) 2015 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+
+/*
+ * pageLock_defs.h
+ *
+ *        Page lock status codes, used by vmmon.
+ */
+
+#ifndef _PAGELOCK_DEFS_H_
+#define _PAGELOCK_DEFS_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_MODULE
+#include "includeCheck.h"
+
+/*
+ * Return codes from page locking, unlocking, and MPN lookup.
+ * They share an error code space because they call one another
+ * internally.
+ *
+ *    PAGE_LOCK_FAILED              The host refused to lock a page.
+ *    PAGE_LOCK_LIMIT_EXCEEDED      We have reached the limit of locked
+ *                                  pages for all VMs
+ *    PAGE_LOCK_TOUCH_FAILED        Failed to touch page after lock.
+ *    PAGE_LOCK_IN_TRANSITION       The page is locked but marked by Windows
+ *                                  as nonpresent in CPU PTE and in transition
+ *                                  in Windows PFN.
+ *
+ *    PAGE_LOCK_SYS_ERROR           System call error.
+ *    PAGE_LOCK_ALREADY_LOCKED      Page already locked.
+ *    PAGE_LOCK_MEMTRACKER_ERROR    MemTracker fails.
+ *    PAGE_LOCK_PHYSTRACKER_ERROR   PhysTracker fails.
+ *    PAGE_LOCK_MDL_ERROR           Mdl error on Windows.
+ *
+ *    PAGE_UNLOCK_NO_ERROR          Unlock successful (must be 0).
+ *    PAGE_UNLOCK_NOT_TRACKED       Not in memtracker.
+ *    PAGE_UNLOCK_NO_MPN            Tracked but no MPN.
+ *    PAGE_UNLOCK_NOT_LOCKED        Not locked.
+ *    PAGE_UNLOCK_TOUCH_FAILED      Failed to touch page.
+ *    PAGE_UNLOCK_MISMATCHED_TYPE   Tracked but was locked by different API
+ *
+ *    PAGE_LOOKUP_INVALID_ADDR      Consistency checking.
+ *    PAGE_LOOKUP_BAD_HIGH_ADDR     Consistency checking.
+ *    PAGE_LOOKUP_ZERO_ADDR         Consistency checking.
+ *    PAGE_LOOKUP_SMALL_ADDR        Consistency checking.
+ *
+ * All error values must be negative values less than -4096 to avoid
+ * conflicts with errno values on Linux.
+ *
+ * -- edward
+ */
+
+#define PAGE_LOCK_SUCCESS                   0
+#define PAGE_LOCK_FAILED              (-10001)
+#define PAGE_LOCK_LIMIT_EXCEEDED      (-10002)
+#define PAGE_LOCK_TOUCH_FAILED        (-10003)
+#define PAGE_LOCK_IN_TRANSITION       (-10004)
+
+#define PAGE_LOCK_SYS_ERROR           (-10010)
+#define PAGE_LOCK_ALREADY_LOCKED      (-10011)
+#define PAGE_LOCK_MEMTRACKER_ERROR    (-10012)
+#define PAGE_LOCK_PHYSTRACKER_ERROR   (-10013)
+#define PAGE_LOCK_MDL_ERROR           (-10014)
+
+#define PAGE_UNLOCK_SUCCESS                 0
+#define PAGE_UNLOCK_NOT_TRACKED       (-10100)
+#define PAGE_UNLOCK_NO_MPN            (-10101)
+#define PAGE_UNLOCK_NOT_LOCKED        (-10102)
+#define PAGE_UNLOCK_TOUCH_FAILED      (-10103)
+#define PAGE_UNLOCK_MISMATCHED_TYPE   (-10104)
+
+#define PAGE_LOOKUP_SUCCESS                 0
+#define PAGE_LOOKUP_INVALID_ADDR      (-10200)
+#define PAGE_LOOKUP_BAD_HIGH_ADDR     (-10201)
+#define PAGE_LOOKUP_ZERO_ADDR         (-10202)
+#define PAGE_LOOKUP_SMALL_ADDR        (-10203)
+#define PAGE_LOOKUP_SYS_ERROR         (-10204)
+#define PAGE_LOOKUP_NOT_TRACKED          (-10)	// added to another code
+#define PAGE_LOOKUP_NO_MPN               (-20)	// added to another code
+#define PAGE_LOOKUP_NOT_LOCKED           (-30)	// added to another code
+#define PAGE_LOOKUP_NO_VM                (-40)	// added to another code
+
+#define PAGE_LOCK_SOFT_FAILURE(status) (status <= PAGE_LOCK_FAILED && \
+                                        status > PAGE_LOCK_SYS_ERROR)
+
+#endif // ifndef _PAGELOCK_DEFS_H_

vmmon-only/include/pagelist.h

@@ -0,0 +1,109 @@
+/*********************************************************
+ * Copyright (C) 2010-2014,2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * pagelist.h -- 
+ *
+ *      Definitions of operations on BPNs used in communicating page info
+ *	between VMKernel/VMX and VMM.
+ */
+
+#ifndef	_PAGELIST_H
+#define	_PAGELIST_H
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_MODULE
+#include "includeCheck.h"
+
+#include "vm_assert.h"
+#include "vmcore_types.h"
+
+#if defined __cplusplus
+extern "C" {
+#endif
+
+
+/*
+ * Sets of pages are passed between the monitor and the platform to be 
+ * shared, invalidated, remapped, or swapped.
+ *
+ * A set is sized so that it fits in a 4KB page.
+ */
+
+#pragma pack(push, 1)
+typedef struct PageListEntry {
+   CompressedBPN cbpn;
+   Bool voided;
+   uint8 _pad[1];
+} PageListEntry;
+#pragma pack(pop) 
+
+#define PAGELIST_MAX     (PAGE_SIZE / sizeof(PageListEntry))
+
+static INLINE void
+PageList_SetEntry(PageListEntry *ple, BPN bpn)
+{
+   CompressedBPN_Write(&ple->cbpn, bpn);
+   ple->voided = FALSE;
+}   
+
+static INLINE BPN
+PageList_BPN(const PageListEntry *ple)
+{
+   return CompressedBPN_Read(&ple->cbpn);
+}
+
+static INLINE Bool
+PageList_IsVoid(const PageListEntry *ple)
+{
+   ASSERT(ple->voided == TRUE || ple->voided == FALSE);
+   return ple->voided;
+}
+
+static INLINE void
+PageList_VoidEntry(PageListEntry *ple)
+{
+   ple->voided = TRUE;
+}
+
+
+/*
+ * This function inspects the set of BPN between entry [0,i) in the page list
+ * and returns TRUE if any of them matches the provided BPN.
+ */
+static INLINE Bool
+PageList_IsBPNDup(const PageListEntry *pageList, unsigned i, BPN bpn)
+{
+   unsigned k;
+   for (k = 0; k < i; k++) {
+      if (PageList_BPN(&pageList[k]) == bpn) {
+         return TRUE;
+      }
+   }
+   return FALSE;
+}
+
+
+#if defined __cplusplus
+} // extern "C"
+#endif
+
+#endif // _PAGELIST_H

vmmon-only/include/pcip_defs.h

@@ -0,0 +1,116 @@
+/*********************************************************
+ * Copyright (C) 2009-2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * pcip_defs.h --
+ *
+ *      PCI passthru definitions shared by the vmx, monitor, vmkernel, and
+ *      vmmon. Not all PCI passthru definitions are found here: the shared
+ *      bits mainly pertain to interrupt proxying.
+ */
+
+#ifndef _PCIP_DEFS_H
+#define _PCIP_DEFS_H
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+#include "monitorAction_exported.h"
+
+#include "bitvector.h"
+
+#define MAX_INTERRUPTS                 256 // max interrupts a device could use
+#define PCIP_MAX_MSIX_VECTORS          128
+#define PCIP_MAX_VECTORS               (PCIP_MAX_MSIX_VECTORS + 2)
+
+typedef struct PCIPVecBV {
+   BitVector bv;
+   uint32 reserved[PCIP_MAX_VECTORS / sizeof (uint32) - 1];
+} PCIPVecBV;
+
+typedef enum PCIPassthruVectorIndex {
+   PCIP_INDEX_IOAPIC,
+   PCIP_INDEX_MSI,
+   PCIP_INDEX_MSIXOFF,
+   PCIP_INDEX_INVALID = PCIP_INDEX_MSIXOFF + PCIP_MAX_MSIX_VECTORS,
+} PCIPassthruVectorIndex;
+
+typedef enum PCIPassthru_IntrType {
+   PCIPASSTHRU_INTR_NONE    = 0x00,
+   PCIPASSTHRU_INTR_IOAPIC  = 0x01,
+   PCIPASSTHRU_INTR_MSI     = 0x02,
+   PCIPASSTHRU_INTR_MSIX    = 0x04,
+   PCIPASSTHRU_INTR_MSI_INT = 0x08,
+} PCIPassthru_IntrType;
+
+#define PCIPASSTHRU_INTR_MSI_COMBINED                   \
+   (PCIPASSTHRU_INTR_MSI | PCIPASSTHRU_INTR_MSI_INT)
+
+typedef struct FPTIntrProxyInfo {
+   uint32            adapterIndex;
+   uint32            vectorIndex;
+   MonitorIdemAction actionID;
+} FPTIntrProxyInfo;
+
+typedef struct UPTIntrProxyInfo {
+   uint32            adapterIndex;
+   MonitorIdemAction actionID;
+} UPTIntrProxyInfo;
+
+typedef union PCIPassthru_IntrProxyInfo {
+   FPTIntrProxyInfo fpt;
+   UPTIntrProxyInfo upt;
+} PCIPassthru_IntrProxyInfo;
+
+typedef enum PCIPassthru_ErrorType {
+   PCIPASSTHRU_ERROR_NONE       = 0x00,
+   PCIPASSTHRU_ERROR_AER        = 0x01,
+   PCIPASSTHRU_ERROR_PAGE_FAULT = 0x02,
+} PCIPassthru_ErrorType;
+
+typedef struct PCIPassthru_PageFaultInfo {
+   uint64 ioAddr;
+   uint64 machAddr;
+   uint8  faultReason;
+   uint8  unused[7];
+} PCIPassthru_PageFaultInfo;
+
+typedef struct PCIPassthru_AERInfo {
+   uint64 count;
+} PCIPassthru_AERInfo;
+
+typedef union PCIPassthru_ErrorInfo {
+   PCIPassthru_PageFaultInfo pageFaultInfo;
+   PCIPassthru_AERInfo aerInfo;
+} PCIPassthru_ErrorInfo;
+
+typedef 
+#include "vmware_pack_begin.h"
+struct PCIPassthru_ErrorMsg {
+   uint32                 sbdf;
+   PCIPassthru_ErrorType  errorType;
+   PCIPassthru_ErrorInfo  errorInfo;
+}
+#include "vmware_pack_end.h"
+PCIPassthru_ErrorMsg;
+
+#endif // _PCIP_DEFS_H

vmmon-only/include/perfctr.h

@@ -0,0 +1,121 @@
+/*********************************************************
+ * Copyright (C) 1998-2012,2014-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * perfctr.h --
+ *
+ */
+
+#ifndef _PERFCTR_H_
+#define _PERFCTR_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+
+#include "includeCheck.h"
+#include "vm_basic_types.h"
+#include "perfctr_arch.h"
+
+#define PERF_EVENT_NAME_LEN                      64
+/*
+ * nmiNo      -- vmm peer is not attempting to do nmi profiling this run
+ * nmiYes     -- vmm peer is doing nmi profiling and nmis are currently enabled
+ * nmiStopped -- vmm peer is doing nmi profiling, but nmis are temporarily
+ *               disabled for safety reasons.
+ */
+typedef enum {nmiNo = 0, nmiYes, nmiStopped} NMIStatus;
+typedef struct NMIShared { /* shared with vmx and vmkernel */
+   NMIStatus vmmStatus;
+   int32     nmiErrorCode;
+   int64     nmiErrorData;
+} NMIShared;
+
+/*
+ * CrossProf: structures for unified profiling of vmm, vmx, and
+ * vmkernel.  Per-vcpu.
+ */
+
+#define CALLSTACK_CROSSPROF_PAGES 1
+
+typedef struct {
+   /*
+    * This structure is per-vcpu.  The raw data is a packed vector
+    * of MonitorCallStackSample, a variable-length structure.
+    */
+
+   /* raw data - packed vec of MonitorCallStackSample, variable length */
+   uint8  crossProfSampleBuffer[PAGES_2_BYTES(CALLSTACK_CROSSPROF_PAGES)];
+
+   uint32 crossProfSampleBytes;
+   uint32 crossProfNumDroppedSamples; /* For when buffer fills up */
+   Bool   enabled; /* Can be false in stats build if monitor.callstack=FALSE */
+   uint8  _pad[3];
+} CrossProfShared;
+
+/*
+ * PerfCtr_Config --
+ *      Describes configuration for a single hardware performance counter
+ *
+ *      Since this is only used to record general performance counters, we
+ *      made the assumption in nmiProfiler.c that the type is GENERAL and
+ *      index is counter number of type GENERAL.
+ *
+ *      **** x86 ****
+ *      On AMD K8 and GH:
+ *      index:        Which perf ctr, 0 to 3.  RDPMC argument
+ *      addr:         MSR of raw perf ctr              (0xc0010004 + index).
+ *      escrAddr:     MSR # of the Perf Event Selector (0xc0010000 + index).
+ *      escrVal:      Value placed in PerfEvtSel MSR; what to measure.
+ *
+ *      On AMD with PerfCtrExtCore support:
+ *      index:        Which perf ctr, 0 to 5.  RDPMC argument
+ *      addr:         MSR of raw perf ctr              (0xc0010201 + 2 * index).
+ *                                  aliased PMCs 0 - 3 (0xc0010004 + index).
+ *      escrAddr:     MSR # of the Perf Event Selector (0xc0010200 + 2 * index).
+ *                                  aliased PMCs 0 - 3 (0xc0010000 + index).
+ *      escrVal:      Value placed in PerfEvtSel MSR; what to measure.
+ *
+ *      On Intel Core architecture:
+ *      <to be documented>
+ *
+ *      **** ARM ****
+ *      escrVal:     Value placed in PMEVTYPER<n>_EL0 to configure event counter
+ *      index:       Index of the event counter.
+ */
+typedef struct PerfCtr_Config {
+   uint64 escrVal;
+   uint32 index;
+   uint32 periodMean;
+
+   /*
+    * Random number (whose absolute value is capped at
+    * periodJitterMask) is used to randomize sampling interval.
+    */
+   uint32  periodJitterMask;
+   uint32  seed;    // seed is used to compute next random number
+   uint16  config;
+   Bool    valid;
+   PERFCTR_CONFIG_ARCH_FIELDS
+} PerfCtr_Config;
+
+
+#endif // ifndef _PERFCTR_H_

vmmon-only/include/perfctr_arch.h

@@ -0,0 +1,751 @@
+/*********************************************************
+ * Copyright (C) 2014, 2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * perfctr_arch.h --
+ *
+ *      Performance counters (x64 specific).
+ */
+
+#ifndef _PERFCTR_ARCH_H
+#define _PERFCTR_ARCH_H
+
+#ifndef _PERFCTR_H_
+#error "This file can only be included by perfctr.h"
+#endif
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+#include "vm_asm.h"
+#include "x86cpuid_asm.h"
+
+#define PERFCTR_AMD_NUM_COUNTERS                 4
+#define PERFCTR_AMD_EXT_NUM_COUNTERS             6
+#define PERFCTR_P6_NUM_COUNTERS                  2
+#define PERFCTR_NEHALEM_NUM_GEN_COUNTERS         4
+#define PERFCTR_NEHALEM_NUM_FIXED_COUNTERS       3
+#define PERFCTR_SANDYBRIDGE_NUM_GEN_COUNTERS     8 /* When HT is disabled */
+#define PERFCTR_CORE_NUM_ARCH_EVENTS             8
+#define PERFCTR_CORE_NUM_FIXED_COUNTERS          4
+#define PERFCTR_AMD_VAL_MASK                     0xffffffffffffLL
+/*
+ * Even though the performance counters in P6 are 40 bits,
+ * we can only write to the lower 32 bits.  Bit 31 is
+ * used to sign-extend the upper 8 bits.
+ */
+#define PERFCTR_P6_VAL_MASK                      0xffffffffffLL
+#define PERFCTR_P6_WRITE_MASK                    0xffffffff
+
+/*
+ * Performance counter width is determined at runtime in CORE.
+ * But the writables bits are fixed. we can only write to the
+ * lower 32 bits.  Bit 31 is used to sign-extend the upper 8 bits.
+ */
+#define PERFCTR_CORE_WRITE_MASK                  0xffffffff
+
+/* Common Event Selection MSR bits. */
+#define PERFCTR_CPU_EVENT_MASK                   0x000000ff
+#define PERFCTR_CPU_EVENT_SHIFT                  0
+#define PERFCTR_CPU_UNIT_MASK                    0x0000ff00
+#define PERFCTR_CPU_UNIT_SHIFT                   8
+#define PERFCTR_CPU_USER_MODE                    0x00010000
+#define PERFCTR_CPU_KERNEL_MODE                  0x00020000
+#define PERFCTR_CPU_EDGE_DETECT                  0x00040000
+#define PERFCTR_CPU_PIN_CONTROL                  0x00080000
+#define PERFCTR_CPU_APIC_INTR                    0x00100000
+#define PERFCTR_CPU_ENABLE                       0x00400000
+#define PERFCTR_CPU_INVERT_COUNTER_MASK          0x00800000
+#define PERFCTR_CPU_COUNTER_MASK                 0xff000000
+#define PERFCTR_CPU_COUNTER_MASK_SHIFT           24
+#define PERFCTR_CPU_EVENT_IN_USE                 0x3C /* Unhalted Core cycles */
+
+/*
+ * ----------------------------------------------------------------------
+ *
+ * AMD K8
+ *
+ * ----------------------------------------------------------------------
+ */
+
+/* AMD Event Selection MSRs */
+#define PERFCTR_AMD_EVENT_MASK                   PERFCTR_CPU_EVENT_MASK
+#define PERFCTR_AMD_EXT_EVENT_MASK               (0xfULL << 32)
+#define PERFCTR_AMD_EVENT_SHIFT                  PERFCTR_CPU_EVENT_SHIFT
+#define PERFCTR_AMD_UNIT_MASK                    PERFCTR_CPU_UNIT_MASK
+#define PERFCTR_AMD_UNIT_SHIFT                   PERFCTR_CPU_UNIT_SHIFT
+#define PERFCTR_AMD_USER_MODE                    PERFCTR_CPU_USER_MODE
+#define PERFCTR_AMD_KERNEL_MODE                  PERFCTR_CPU_KERNEL_MODE
+#define PERFCTR_AMD_EDGE_DETECT                  PERFCTR_CPU_EDGE_DETECT
+#define PERFCTR_AMD_PIN_CONTROL                  PERFCTR_CPU_PIN_CONTROL
+#define PERFCTR_AMD_APIC_INTR                    PERFCTR_CPU_APIC_INTR
+#define PERFCTR_AMD_ENABLE                       PERFCTR_CPU_ENABLE
+#define PERFCTR_AMD_INVERT_COUNTER_MASK          PERFCTR_CPU_INVERT_COUNTER_MASK
+#define PERFCTR_AMD_COUNTER_MASK                 PERFCTR_CPU_COUNTER_MASK
+#define PERFCTR_AMD_COUNTER_MASK_SHIFT           PERFCTR_CPU_COUNTER_MASK_SHIFT
+#define PERFCTR_AMD_SHIFT_BY_UNITMASK(e)         ((e) << 8 )
+#define PERFCTR_AMD_EVTSEL_HOST                  (CONST64U(1) << 41)
+#define PERFCTR_AMD_EVTSEL_GUEST                 (CONST64U(1) << 40)
+
+/* AMD Performance Counter MSR Definitions */
+#define PERFCTR_AMD_PERFEVTSEL0_ADDR             0xC0010000
+#define PERFCTR_AMD_PERFCTR0_ADDR                0xC0010004
+/* AMD with PerfCtrExtCore (PERFCORE) support MSR Definitions */
+#define PERFCTR_AMD_EXT_BASE_ADDR                0xC0010200
+#define PERFCTR_AMD_EXT_EVENTSEL                 0
+#define PERFCTR_AMD_EXT_CTR                      1
+#define PERFCTR_AMD_EXT_MSR_STRIDE               2
+#define PERFCTR_AMD_AMD_EXT_CNTR_BASE_ADDR \
+        (PERFCTR_AMD_EXT_BASE_ADDR + PERFCTR_AMD_EXT_CTR)
+#define PERFCTR_AMD_AMD_EXT_EVSL_BASE_ADDR \
+        (PERFCTR_AMD_EXT_BASE_ADDR + PERFCTR_AMD_EXT_EVENTSEL)
+
+/* AMD Clocks */
+#define PERFCTR_AMD_CPU_CLK_UNHALTED                           0x76
+
+/* AMD Load/Store unit events */
+#define PERFCTR_AMD_SEGMENT_REGISTER_LOADS                     0x20
+#define PERFCTR_AMD_LS_BUFFER2_FULL                            0x23
+
+/*
+ * Event 0x2b counts SMIs on Opteron Rev.G (with upatch) and on
+ * GH >= Rev.B0 without patches. GH Rev.A has no such capability.
+ */
+#define PERFCTR_AMD_SMI_COUNT                                  0x2b
+
+/* AMD Data Cache Events */
+#define PERFCTR_AMD_DATA_CACHE_ACCESSES                        0x40
+#define PERFCTR_AMD_DATA_CACHE_MISSES                          0x41
+#define PERFCTR_AMD_DATA_CACHE_REFILLS_FROM_L2_OR_SYSTEM       0x42
+#define PERFCTR_AMD_DATA_CACHE_REFILLS_FROM_SYSTEM             0x43
+#define PERFCTR_AMD_DATA_CACHE_LINES_EVICTED                   0x44
+#define PERFCTR_AMD_L1_DTLB_MISS_AND_L2_DTLB_HIT               0x45
+#define PERFCTR_AMD_L1_DTLB_AND_L2_DTLB_MISS                   0x46
+#define PERFCTR_AMD_MISALIGNED_ACCESSES                        0x47
+#define PERFCTR_AMD_PREFETCH_INSTRS_DISPATCHED                 0x4b
+#define PERFCTR_AMD_DCACHE_MISSES_BY_LOCKED_INSTR              0x4c
+
+/* AMD L2 Cache */
+#define PERFCTR_AMD_REQUESTS_TO_L2                             0x7d
+#define PERFCTR_AMD_L2_MISS                                    0x7e
+#define PERFCTR_AMD_L2_FILL_WRITEBACK                          0x7f
+
+/* AMD Instruction Cache Events */
+
+#define PERFCTR_AMD_INSTR_FETCHES                              0x80
+#define PERFCTR_AMD_INSTR_MISSES                               0x81
+#define PERFCTR_AMD_INSTR_REFILLS_FROM_L2                      0x82
+#define PERFCTR_AMD_INSTR_REFILLS_FROM_SYSTEM                  0x83
+#define PERFCTR_AMD_L1_ITLB_MISS_L2_ITLB_HIT                   0x84
+#define PERFCTR_AMD_L1_ITLB_MISS_L2_ITLB_MISS                  0x85
+#define PERFCTR_AMD_INSTR_FETCH_STALL                          0x87
+
+/* AMD Execution Unit Events */
+#define PERFCTR_AMD_RET_INSTR                                  0xc0
+#define PERFCTR_AMD_RET_UOPS                                   0xc1
+#define PERFCTR_AMD_RET_BRANCH_INSTR                           0xc2
+#define PERFCTR_AMD_RET_MISPRED_BRANCH_INSTR                   0xc3
+#define PERFCTR_AMD_RET_TAKEN_BRANCH_INSTR                     0xc4
+#define PERFCTR_AMD_RET_TAKEN_BRANCH_INSTR_MISPRED             0xc5
+#define PERFCTR_AMD_RET_FAR_CONTROL_TRANSFERS                  0xc6
+#define PERFCTR_AMD_RET_BRANCH_RESYNCS                         0xc7
+#define PERFCTR_AMD_RET_NEAR_RETURNS                           0xc8
+#define PERFCTR_AMD_RET_NEAR_RETURNS_MISPRED                   0xc9
+#define PERFCTR_AMD_RET_INDIRECT_BRANCHES_MISPRED              0xca
+#define PERFCTR_AMD_RET_MMX_FP_INSTR                           0xcb
+#define PERFCTR_AMD_INT_MASKED_CYCLES                          0xcd
+#define PERFCTR_AMD_INT_MASKED_CYCLES_WITH_INT_PEND            0xce
+
+#define PERFCTR_AMD_INT_MASKED_COUNT \
+  ((0xcd) | PERFCTR_AMD_EDGE_DETECT)
+
+#define PERFCTR_AMD_INT_MASKED_COUNT_WITH_INT_PEND \
+  ((0xce) | PERFCTR_AMD_EDGE_DETECT)
+
+#define PERFCTR_AMD_INT_TAKEN                                       0xcf
+#define PERFCTR_AMD_DECODER_EMPTY_CYCLES                            0xd0
+#define PERFCTR_AMD_DISPATCH_STALLS                                 0xd1
+#define PERFCTR_AMD_DISPATCH_MISPRED_BRANCH_STALL_CYCLES            0xd2
+#define PERFCTR_AMD_DISPATCH_SERIALIZATION_STALL_CYCLES             0xd3
+#define PERFCTR_AMD_DISPATCH_SEGMENTLOAD_STALL_CYCLES               0xd4
+#define PERFCTR_AMD_DISPATCH_REORDER_BUFFER_FULL_STALL_CYCLES       0xd5
+#define PERFCTR_AMD_DISPATCH_RESERVATION_STATION_FULL_STALL_CYCLES  0xd6
+#define PERFCTR_AMD_DISPATCH_LS_FULL_STALL_CYCLES                   0xd8
+#define PERFCTR_AMD_DISPATCH_WAIT_ALLQUIET_STALL_CYCLES             0xd9
+#define PERFCTR_AMD_DISPATCH_FAR_XFER_OR_RESYNC_RETIRE_STALL_CYCLES 0xda
+
+/* AMD Memory Controller Events */
+
+#define PERFCTR_AMD_MEM_CTRL_PAGE_TABLE_OVERFLOWS                   0xe1
+#define PERFCTR_AMD_CPU_IO_REQUESTS_TO_MEMORY_IO                    0xe9
+#define PERFCTR_AMD_PROBE_RESPONSE_AND_UPSTREAM_REQ                 0xec
+
+/* AMD HyperTransport Interface Events */
+
+#define PERFCTR_AMD_HT_L0_TX_BW                                     0xf6
+#define PERFCTR_AMD_HT_L1_TX_BW                                     0xf7
+#define PERFCTR_AMD_HT_L2_TX_BW                                     0xf8
+
+/*
+ * ----------------------------------------------------------------------
+ *
+ * Intel P6 family (excluding newer Core architecture)
+ *
+ * ----------------------------------------------------------------------
+ */
+
+/*
+ * Event numbers for P6 Family Processors.
+ */
+
+/* P6 Data Cache Unit (DCU) */
+#define PERFCTR_P6_DATA_MEM_REFS                 0x00000043
+#define PERFCTR_P6_DCU_LINES_IN                  0x00000045
+#define PERFCTR_P6_DCU_M_LINES_IN                0x00000046
+#define PERFCTR_P6_DCU_MISS_OUTSTANDING          0x00000048
+
+/* P6 Instruction Fetch Unit (IFU) */
+#define PERFCTR_P6_IFU_IFETCH                    0x00000080
+#define PERFCTR_P6_IFU_IFETCH_MISS               0x00000081
+#define PERFCTR_P6_ITLB_MISS                     0x00000085
+#define PERFCTR_P6_IFU_MEM_STALL                 0x00000086
+#define PERFCTR_P6_ILD_STALL                     0x00000087
+
+/* P6 L2 Cache */
+#define PERFCTR_P6_L2_IFETCH                     0x00000f28
+#define PERFCTR_P6_L2_LD                         0x00000f29
+#define PERFCTR_P6_L2_ST                         0x00000f2a
+#define PERFCTR_P6_L2_LINES_IN                   0x00000024
+#define PERFCTR_P6_L2_LINES_OUT                  0x00000026
+#define PERFCTR_P6_L2_LINES_INM                  0x00000025
+#define PERFCTR_P6_L2_LINES_OUTM                 0x00000027
+#define PERFCTR_P6_L2_RQSTS                      0x00000f2e
+#define PERFCTR_P6_L2_ADS                        0x00000021
+#define PERFCTR_P6_L2_DBUS_BUSY_RD               0x00000023
+
+/* P6 External Bus Logic */
+#define PERFCTR_P6_BUSDRDY_CLOCKS_SELF           0x00000062
+#define PERFCTR_P6_BUSDRDY_CLOCKS_ANY            0x00002062
+#define PERFCTR_P6_BUS_LOCK_CLOCKS_SELF          0x00000063
+#define PERFCTR_P6_BUS_LOCK_CLOCKS_ANY           0x00002063
+#define PERFCTR_P6_BUS_REQ_OUTSTANDING           0x00000060
+#define PERFCTR_P6_BUS_TRAN_BRD_SELF             0x00000065
+#define PERFCTR_P6_BUS_TRAN_BRD_ANY              0x00002065
+#define PERFCTR_P6_BUS_TRAN_RFO_SELF             0x00000066
+#define PERFCTR_P6_BUS_TRAN_RFO_ANY              0x00002066
+#define PERFCTR_P6_BUS_TRAN_WB_SELF              0x00000067
+#define PERFCTR_P6_BUS_TRAN_WB_ANY               0x00002067
+#define PERFCTR_P6_BUS_TRAN_IFETCH_SELF          0x00000068
+#define PERFCTR_P6_BUS_TRAN_IFETCH_ANY           0x00002068
+#define PERFCTR_P6_BUS_TRAN_INVAL_SELF           0x00000069
+#define PERFCTR_P6_BUS_TRAN_INVAL_ANY            0x00002069
+#define PERFCTR_P6_BUS_TRAN_PWR_SELF             0x0000006a
+#define PERFCTR_P6_BUS_TRAN_PWR_ANY              0x0000206a
+#define PERFCTR_P6_BUS_TRAN_P_SELF               0x0000006b
+#define PERFCTR_P6_BUS_TRAN_P_ANY                0x0000206b
+#define PERFCTR_P6_BUS_TRAN_IO_SELF              0x0000006c
+#define PERFCTR_P6_BUS_TRAN_IO_ANY               0x0000206c
+#define PERFCTR_P6_BUS_TRAN_DEF_SELF             0x0000006d
+#define PERFCTR_P6_BUS_TRAN_DEF_ANY              0x0000206d
+#define PERFCTR_P6_BUS_TRAN_DEF_SELF             0x0000006d
+#define PERFCTR_P6_BUS_TRAN_DEF_ANY              0x0000206d
+#define PERFCTR_P6_BUS_TRAN_BURST_SELF           0x0000006e
+#define PERFCTR_P6_BUS_TRAN_BURST_ANY            0x0000206e
+#define PERFCTR_P6_BUS_TRAN_ANY_SELF             0x00000070
+#define PERFCTR_P6_BUS_TRAN_ANY_ANY              0x00002070
+#define PERFCTR_P6_BUS_TRAN_MEM_SELF             0x0000006f
+#define PERFCTR_P6_BUS_TRAN_MEM_ANY              0x0000206f
+#define PERFCTR_P6_BUS_TRAN_RCV                  0x00000064
+#define PERFCTR_P6_BUS_BNR_DRV                   0x00000061
+#define PERFCTR_P6_BUS_HIT_DRV                   0x0000007a
+#define PERFCTR_P6_BUS_HITM_DRV                  0x0000007b
+#define PERFCTR_P6_BUS_SNOOP_STALL               0x0000007E
+
+/* P6 Floating-Point Unit */
+#define PERFCTR_P6_FLOPS                         0x000000c1
+#define PERFCTR_P6_FP_COMP_OPS_EXE               0x00000010
+#define PERFCTR_P6_FP_ASSIST                     0x00000011
+#define PERFCTR_P6_MUL                           0x00000012
+#define PERFCTR_P6_DIV                           0x00000013
+#define PERFCTR_P6_CYCLES_DIV_BUSY               0x00000014
+
+/* P6 Memory Ordering */
+#define PERFCTR_P6_LD_BLOCKS                     0x00000003
+#define PERFCTR_P6_SB_DRAINS                     0x00000004
+#define PERFCTR_P6_MISALIGN_MEM_REF              0x00000005
+#define PERFCTR_P6_EMON_KNI_PREF_DISPATCHED_NTA  0x00000007
+#define PERFCTR_P6_EMON_KNI_PREF_DISPATCHED_T1   0x00000107
+#define PERFCTR_P6_EMON_KNI_PREF_DISPATCHED_T2   0x00000207
+#define PERFCTR_P6_EMON_KNI_PREF_DISPATCHED_WOS  0x00000307
+#define PERFCTR_P6_EMON_KNI_PREF_MISS_NTA        0x0000004b
+#define PERFCTR_P6_EMON_KNI_PREF_MISS_T1         0x0000014b
+#define PERFCTR_P6_EMON_KNI_PREF_MISS_T2         0x0000024b
+#define PERFCTR_P6_EMON_KNI_PREF_MISS_WOS        0x0000034b
+
+/* P6 Instruction Decoding and Retirement */
+#define PERFCTR_P6_INST_RETIRED                  0x000000c0
+#define PERFCTR_P6_UOPS_RETIRED                  0x000000c2
+#define PERFCTR_P6_INST_DECODED                  0x000000d0
+#define PERFCTR_P6_EMON_KNI_INST_RETIRED         0x000000d8
+#define PERFCTR_P6_EMON_KNI_INST_RETIRED_SC      0x000001d8
+#define PERFCTR_P6_EMON_KNI_COMP_INST_RETIRED    0x000000d9
+#define PERFCTR_P6_EMON_KNI_COMP_INST_RETIRED_SC 0x000001d9
+
+/* P6 Interrupts */
+#define PERFCTR_P6_HW_INT_RX                     0x000000c8
+#define PERFCTR_P6_CYCLES_INT_MASKED             0x000000c6
+#define PERFCTR_P6_CYCLES_INT_PENDING_AND_MASKED 0x000000c7
+
+/* P6 Branches */
+#define PERFCTR_P6_BR_INST_RETIRED               0x000000c4
+#define PERFCTR_P6_BR_MISS_PRED_RETIRED          0x000000c5
+#define PERFCTR_P6_BR_TAKEN_RETIRED              0x000000c9
+#define PERFCTR_P6_BR_MISS_PRED_TAKEN_RET        0x000000ca
+#define PERFCTR_P6_BR_INST_DECODED               0x000000e0
+#define PERFCTR_P6_BTB_MISSES                    0x000000e2
+#define PERFCTR_P6_BR_BOGUS                      0x000000e4
+#define PERFCTR_P6_BACLEARS                      0x000000e6
+
+/* P6 Stalls */
+#define PERFCTR_P6_RESOURCE_STALLS               0x000000a2
+#define PERFCTR_P6_PARTIAL_RAT_CLEARS            0x000000d2
+
+/* P6 Segment Register Loads */
+#define PERFCTR_P6_SEGMENT_REG_LOADS             0x00000006
+
+/* P6 Clocks */
+#define PERFCTR_P6_CPU_CLK_UNHALTED              0x00000079
+
+/* P6 MMX Unit */
+#define PERFCTR_P6_MMX_INSTR_EXEC                0x000000b0
+#define PERFCTR_P6_MMX_SAT_INSTR_EXEC            0x000000b1
+#define PERFCTR_P6_MMX_UOPS_EXEC                 0x000000b2
+#define PERFCTR_P6_MMX_INSTR_TYPE_EXEC_PK_MUL    0x000001b3
+#define PERFCTR_P6_MMX_INSTR_TYPE_EXEC_PK_SHIFT  0x000002b3
+#define PERFCTR_P6_MMX_INSTR_TYPE_EXEC_PK_OP     0x000004b3
+#define PERFCTR_P6_MMX_INSTR_TYPE_EXEC_UNPK_OP   0x000008b3
+#define PERFCTR_P6_MMX_INSTR_TYPE_EXEC_PK_LOG    0x000010b3
+#define PERFCTR_P6_MMX_INSTR_TYPE_EXEC_PK_ARITH  0x000020b3
+#define PERFCTR_P6_FP_MMX_TRANS_TO               0x000000cc
+#define PERFCTR_P6_FP_MMX_TRANS_FROM             0x000001cc
+#define PERFCTR_P6_FP_MMX_ASSIST                 0x000000cd
+#define PERFCTR_P6_FP_MMX_INSTR_RET              0x000000ce
+
+/* P6 Segment Register Renaming */
+#define PERFCTR_P6_SEG_RENAME_STALLS_ES          0x000001d4
+#define PERFCTR_P6_SEG_RENAME_STALLS_DS          0x000002d4
+#define PERFCTR_P6_SEG_RENAME_STALLS_FS          0x000004d4
+#define PERFCTR_P6_SEG_RENAME_STALLS_GS          0x000008d4
+#define PERFCTR_P6_SEG_RENAME_STALLS_ANY         0x00000fd4
+#define PERFCTR_P6_SEG_RENAMES_ES                0x000001d5
+#define PERFCTR_P6_SEG_RENAMES_DS                0x000002d5
+#define PERFCTR_P6_SEG_RENAMES_FS                0x000004d5
+#define PERFCTR_P6_SEG_RENAMES_GS                0x000008d5
+#define PERFCTR_P6_SEG_RENAMES_ANY               0x00000fd5
+#define PERFCTR_P6_RET_SEG_RENAMES               0x000000d6
+
+/*
+ * P6 Event Selection MSRs
+ */
+
+#define PERFCTR_P6_EVENT_MASK                    0x000000ff
+#define PERFCTR_P6_EVENT_SHIFT                   0
+#define PERFCTR_P6_UNIT_MASK                     0x0000ff00
+#define PERFCTR_P6_UNIT_SHIFT                    8
+#define PERFCTR_P6_USER_MODE                     0x00010000
+#define PERFCTR_P6_KERNEL_MODE                   0x00020000
+#define PERFCTR_P6_EDGE_DETECT                   0x00040000
+#define PERFCTR_P6_PIN_CONTROL                   0x00080000
+#define PERFCTR_P6_APIC_INTR                     0x00100000
+#define PERFCTR_P6_ENABLE                        0x00400000
+#define PERFCTR_P6_INVERT_COUNTER_MASK           0x00800000
+#define PERFCTR_P6_COUNTER_MASK                  0xff000000
+#define PERFCTR_P6_COUNTER_MASK_SHIFT            24
+#define PERFCTR_P6_SHIFT_BY_UNITMASK(e)          (e << 8)
+
+/*
+ * P6 Performance Counter MSR Addresses
+ */
+#define PERFCTR_P6_PERFEVTSEL0_ADDR              0x00000186
+#define PERFCTR_P6_PERFCTR0_ADDR                 0x000000c1
+
+/*
+ * ----------------------------------------------------------------------
+ *
+ * Intel Core architecture
+ *
+ *    Use CPUID 0xa to get perf capabilities.  Some (7) events are
+ *    architectural; most are version-specific.
+ *
+ *    V1: Yonah, V2: Merom, V2+: Penryn, V3: Nehalem, V4: Skylake
+ *
+ *    V1 is similar to P6, with some additions:
+ *       - Global control MSR
+ *    V2 introduces:
+ *       - 3 fixed counters
+ *       - Ability to freeze a counter before PMI delivery.
+ *       - Freeze during SMI (Penryn+ only)
+ *       - VMCS global enable (Penryn+ only)
+ *    V3 introduces:
+ *       - (nothing we virtualize)
+ *    V4 introduces:
+ *       - Global unavailable (in-use) MSR
+ *       - Freeze on PMI bit
+ *       - ASCI bit
+ *       - Global status set MSR
+ *       - Global status reset MSR (previously called Ovf ctrl)
+ *
+ * ----------------------------------------------------------------------
+ */
+#define PERFCTR_CORE_PERFCTR0_ADDR                  0x0c1
+#define PERFCTR_CORE_PERFEVTSEL0_ADDR               0x186
+#define PERFCTR_CORE_FIXED_CTR0_ADDR                0x309
+#define PERFCTR_CORE_FIXED_CTR_CTRL_ADDR            0x38d
+#define PERFCTR_CORE_FIXED_CTR_CTRL_PMI_MASK        0x888
+#define PERFCTR_CORE_GLOBAL_STATUS_ADDR             0x38e
+#define PERFCTR_CORE_GLOBAL_CTRL_ADDR               0x38f
+#define PERFCTR_CORE_GLOBAL_OVF_CTRL_ADDR           0x390
+#define PERFCTR_CORE_GLOBAL_STATUS_RESET_ADDR       0x390
+#define PERFCTR_CORE_GLOBAL_STATUS_SET_ADDR         0x391
+#define PERFCTR_CORE_GLOBAL_UNAVAILABLE_STATUS_ADDR 0x392
+#define PERFCTR_CORE_PERFCTR0_FULL_WIDTH_ADDR       0x4c1
+#define PERFCTR_CORE_GLOBAL_PMC0_ENABLE             0x1
+#define PERFCTR_CORE_GLOBAL_PMC1_ENABLE             0x2
+#define PERFCTR_CORE_GLOBAL_FIXED_ENABLE            0x700000000ULL
+#define PERFCTR_CORE_USER_MODE                      PERFCTR_CPU_USER_MODE
+#define PERFCTR_CORE_KERNEL_MODE                    PERFCTR_CPU_KERNEL_MODE
+#define PERFCTR_CORE_APIC_INTR                      PERFCTR_CPU_APIC_INTR
+#define PERFCTR_CORE_ENABLE                         PERFCTR_CPU_ENABLE
+/* AnyThread Counting deprecated since PMU v5. */
+#define PERFCTR_CORE_ANYTHREAD                      0x00200000
+#define PERFCTR_CORE_IN_TX                          (CONST64U(1) << 32)
+#define PERFCTR_CORE_IN_TXCP                        (CONST64U(1) << 33)
+#define PERFCTR_CORE_SHIFT_BY_UNITMASK(e)           ((e) << 8)
+#define PERFCTR_CORE_FIXED_CTR0_PMC                 0x40000000
+#define PERFCTR_CORE_FIXED_CTR1_PMC                 0x40000001
+#define PERFCTR_CORE_FIXED_PMI_MASKn(n)             (CONST64U(0x8) << ((n) * 4))
+#define PERFCTR_CORE_FIXED_ANY_MASKn(n)             (CONST64U(0x4) << ((n) * 4))
+#define PERFCTR_CORE_FIXED_KERNEL_MASKn(n)          (CONST64U(0x1) << ((n) * 4))
+#define PERFCTR_CORE_FIXED_USER_MASKn(n)            (CONST64U(0x2) << ((n) * 4))
+#define PERFCTR_CORE_FIXED_ENABLE_MASKn(n)          (CONST64U(0x3) << ((n) * 4))
+#define PERFCTR_CORE_FIXED_MASKn(n)                 (CONST64U(0xf) << ((n) * 4))
+#define PERFCTR_CORE_FIXED_SHIFTBYn(n)              ((n) * 4)
+#define PERFCTR_CORE_FIXED_ANYTHREAD                CONST64U(0x00000444)
+#define PERFCTR_CORE_PMI_UNAVAILABLE_IN_USE         (CONST64U(1) << 63)
+// XXX serebrin/dhecht: 1-10-11: Make ANYTHREAD depend on number of fixed PMCs
+
+#define PERFCTR_CORE_GLOBAL_STATUS_TOPA_PMI      (1ULL << 55)
+#define PERFCTR_CORE_GLOBAL_STATUS_CTR_FRZ       (1ULL << 59)
+#define PERFCTR_CORE_GLOBAL_STATUS_ASCI          (1ULL << 60)
+#define PERFCTR_CORE_GLOBAL_STATUS_OVFBUFFER     (1ULL << 62)
+
+/* Architectural event counters */
+#define PERFCTR_CORE_UNHALTED_CORE_CYCLES        0x3c
+#define PERFCTR_CORE_INST_RETIRED                0xc0
+/* bus cycles */
+#define PERFCTR_CORE_UNHALTED_REF_CYCLES         (0x3c | (0x01 << 8))
+#define PERFCTR_CORE_TOPDOWN_SLOTS               (0xa4 | (0x01 << 8))
+
+/*
+ * See Tables 30-2, 30-4 of the
+ * "Intel 64 and IA-32 Architecture's Software Developer Manual,
+ * Volume 3B, System Programming Guide, Part 2
+ */
+#define PERFCTR_CORE_LLC_REF                     (0x2e | (0x4f << 8))
+#define PERFCTR_CORE_LLC_MISSES                  (0x2e | (0x41 << 8))
+#define PERFCTR_CORE_LLC_MISSES_PREFETCH         (0x2e | (0x71 << 8))
+#define PERFCTR_CORE_LLC_MISSES_ALL              (0x2e | (0xc1 << 8))
+#define PERFCTR_CORE_LLC_MISSES_ALL_PREFETCH     (0x2e | (0xf1 << 8))
+#define PERFCTR_CORE_BRANCH_RETIRED              0xc4
+#define PERFCTR_CORE_BRANCH_MISPRED_RETIRED      0xc5
+
+/* Non-Architectural event counters in Intel Core and Core 2 */
+#define PERFCTR_CORE_L2_LINES_IN                 0x24
+#define PERFCTR_CORE_L2_M_LINES_IN               0x25
+#define PERFCTR_CORE_L2_LINES_OUT                0x26
+#define PERFCTR_CORE_L2_M_LINES_OUT              0x27
+#define PERFCTR_CORE_DATA_MEM_REF                0x43
+#define PERFCTR_CORE_DATA_MEM_CACHE_REF          0x44
+#define PERFCTR_CORE_DCACHE_REPL                 0x45
+#define PERFCTR_CORE_DCACHE_M_REPL               0x46
+#define PERFCTR_CORE_DCACHE_M_EVICT              0x47
+#define PERFCTR_CORE_DCACHE_PEND_MISS            0x48
+#define PERFCTR_CORE_DTLB_MISS                   0x49
+#define PERFCTR_CORE_BUS_TRANS                   0x70
+#define PERFCTR_CORE_ICACHE_READS                0x80
+#define PERFCTR_CORE_ICACHE_MISSES               0x81
+#define PERFCTR_CORE_ITLB_MISSES                 0x85
+#define PERFCTR_CORE_UOPS_RETIRED                0xC2
+#define PERFCTR_CORE_RESOURCE_STALLS             0xDC
+#define PERFCTR_NEHALEM_OFFCORE_RESP0_EVENT      (0xB7 | (0x01 << 8))
+#define PERFCTR_NEHALEM_OFFCORE_RESP1_EVENT      (0xBB | (0x01 << 8))
+
+/* Intel TSX performance events introduced on Haswell */
+#define PERFCTR_HASWELL_HLE_RETIRED_START        (0xc8 | (0x01 << 8))
+#define PERFCTR_HASWELL_HLE_RETIRED_COMMIT       (0xc8 | (0x02 << 8))
+#define PERFCTR_HASWELL_HLE_RETIRED_ABORT        (0xc8 | (0x04 << 8))
+#define PERFCTR_HASWELL_RTM_RETIRED_START        (0xc9 | (0x01 << 8))
+#define PERFCTR_HASWELL_RTM_RETIRED_COMMIT       (0xc9 | (0x02 << 8))
+#define PERFCTR_HASWELL_RTM_RETIRED_ABORT        (0xc9 | (0x04 << 8))
+
+/*
+ * Nehalem off-core response events. See Section 30.6.1.2 of the
+ * "Intel 64 and IA-32 Architecture's Software Developer Manual,
+ * Volume 3B, System Programming Guide, Part 2.
+ * One can specify
+ * (request from the core, response from the uncore) pairs.
+ */
+#define PERFCTR_NEHALEM_OFFCORE_RESP0_ADDR             0x1A6
+// requests
+#define PERFCTR_NEHALEM_OFFCORE_RQST_DMND_DATA_RD       0x1
+#define PERFCTR_NEHALEM_OFFCORE_RQST_DMND_RFO           0x2
+#define PERFCTR_NEHALEM_OFFCORE_RQST_DMND_IFETCH        0x4
+#define PERFCTR_NEHALEM_OFFCORE_RQST_WB                 0x8
+#define PERFCTR_NEHALEM_OFFCORE_RQST_PF_DATA_RD         0x10
+#define PERFCTR_NEHALEM_OFFCORE_RQST_PF_RFO             0x20
+#define PERFCTR_NEHALEM_OFFCORE_RQST_PF_IFETCH          0x40
+#define PERFCTR_NEHALEM_OFFCORE_RQST_OTHER              0x80
+
+// responses
+#define PERFCTR_NEHALEM_OFFCORE_RESP_UNCORE_HIT         0x100
+#define PERFCTR_NEHALEM_OFFCORE_RESP_OTHER_CORE_HIT_SNP 0x200
+#define PERFCTR_NEHALEM_OFFCORE_RESP_OTHER_CORE_HITM    0x400
+#define PERFCTR_NEHALEM_OFFCORE_RESP_REMOTE_CACHE_FWD   0x1000
+#define PERFCTR_NEHALEM_OFFCORE_RESP_REMOTE_DRAM        0x2000
+#define PERFCTR_NEHALEM_OFFCORE_RESP_LOCAL_DRAM         0x4000
+#define PERFCTR_NEHALEM_OFFCORE_RESP_NON_DRAM           0x8000
+
+// Nehalem Uncore events.
+
+/*
+ * Uncore event MSRs. See table B-5 "MSRs in processors based on the
+ * Intel micro-architecture (Nehalem), in "Intel 64 and IA-32 Architecture's
+ * Software Developer Manual, Volume 3B, System Programming Guide, Part 2".
+ * Though note that the addresses listed for PMCs and event select MSRs are
+ * incorrect (swapped) in the manual.
+ */
+#define PERFCTR_NEHALEM_UNCORE_GLOBALCTRL_ADDR   0x391
+#define PERFCTR_NEHALEM_UNCORE_PERFEVTSEL0_ADDR  0x3c0
+#define PERFCTR_NEHALEM_UNCORE_PERFCTR0_ADDR     0x3b0
+
+/*
+ * The uncore event masks. See section 30.6.2 of the
+ * "Intel 64 and IA-32 Architecture's Software Developer Manual,
+ * Volume 3B, System Programming Guide, Part 2"
+ */
+
+// Enable the use of the programmable uncore counter "x".
+#define PERFCTR_NEHALEM_UNCORE_ENABLE_CTR(x)     (1 << (x))
+#define PERFCTR_NEHALEM_UNCORE_EDGE_DETECT       0x40000
+#define PERFCTR_NEHALEM_UNCORE_ENABLE            0x400000
+#define PERFCTR_NEHALEM_UNCORE_L3_LINES_IN       (0x0a | (0x0f << 8))
+
+#define PERFCTR_CONFIG_ARCH_FIELDS \
+   uint32 addr;      \
+   uint32 escrAddr;  \
+   uint32 resetHi;   \
+   Bool pebsEnabled; \
+
+/*
+ * Program/reprogram event reg(s) associated w/perfctrs & start or stop perfctrs
+ */
+static INLINE void
+PerfCtr_WriteEvtSel(uint32 addr,       // IN: Event register to write
+                    uint32 escrVal)    // IN: event register value
+{
+   X86MSR_SetMSR(addr, escrVal);
+}
+
+
+/*
+ * Set/reset performance counters to engender desired period before overflow
+ */
+static INLINE void
+PerfCtr_WriteCounter(uint32 addr,   // IN: counter to write
+                     uint64 value)  // IN: value to write
+{
+   X86MSR_SetMSR(addr, value);
+}
+
+
+static INLINE uint64
+PerfCtr_SelValidBits(Bool amd)
+{
+   /*
+    * Intel enforces PIN_CONTROL as MBZ; AMD does not.  Always mask on AMD
+    * to avoid toggling the physical pin.
+    */
+   uint64 bits = PERFCTR_CPU_EVENT_MASK  | PERFCTR_CPU_UNIT_MASK |
+                 PERFCTR_CPU_USER_MODE   | PERFCTR_CPU_KERNEL_MODE |
+                 PERFCTR_CPU_EDGE_DETECT | PERFCTR_CPU_APIC_INTR |
+                 PERFCTR_CPU_ENABLE      | PERFCTR_CPU_INVERT_COUNTER_MASK |
+                 PERFCTR_CPU_COUNTER_MASK;
+   if (amd) {
+      bits |= PERFCTR_AMD_EXT_EVENT_MASK | PERFCTR_AMD_EVTSEL_HOST |
+              PERFCTR_AMD_EVTSEL_GUEST;
+   } else {
+      bits |= PERFCTR_CORE_ANYTHREAD | PERFCTR_CORE_IN_TX |
+              PERFCTR_CORE_IN_TXCP;
+   }
+   return bits;
+}
+
+static INLINE uint64
+PerfCtr_PgcValidBits(unsigned numGenCtrs, unsigned numFixCtrs)
+{
+   return MASK64(numGenCtrs) | (MASK64(numFixCtrs) << 32);
+}
+
+static INLINE uint64
+PerfCtr_FccValidBits(unsigned numFixCtrs)
+{
+   return MASK64(numFixCtrs * 4);
+}
+
+static INLINE uint64
+PerfCtr_PgcToOvfValidBits(uint64 pgcValBits)
+{
+   return pgcValBits | MASKRANGE64(63, 61);
+}
+
+static INLINE uint64
+PerfCtr_PgcToStsRstValidBits(uint64 pgcValBits)
+{
+   return pgcValBits |
+          PERFCTR_CORE_GLOBAL_STATUS_TOPA_PMI |
+          MASKRANGE64(63, 58);
+}
+
+static INLINE uint64
+PerfCtr_PgcToGssValidBits(uint64 pgcValBits)
+{
+   return pgcValBits |
+          PERFCTR_CORE_GLOBAL_STATUS_TOPA_PMI |
+          MASKRANGE64(62, 58);
+}
+
+/*
+ *----------------------------------------------------------------------
+ *
+ *  PerfCtr_HypervisorCPUIDSig --
+ *
+ *      Get the hypervisor signature string from CPUID.
+ *
+ * Results:
+ *      TRUE on success FALSE otherwise.
+ *      Unqualified 16 byte nul-terminated hypervisor string which may contain
+ *      garbage.
+ *
+ * Side effects:
+ *      None
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+PerfCtr_HypervisorCPUIDSig(CPUIDRegs *name)
+{
+   CPUIDRegs regs;
+
+   __GET_CPUID(1, &regs);
+   if (!CPUID_ISSET(1, ECX, HYPERVISOR, regs.ecx)) {
+      return FALSE;
+   }
+
+   __GET_CPUID(0x40000000, name);
+
+   if (name->eax < 0x40000000) {
+      return FALSE;
+   }
+
+   return TRUE;
+}
+
+/*
+ *----------------------------------------------------------------------
+ *
+ *  PerfCtr_PEBSAvailable --
+ *
+ *      Checks if this CPU is capable of PEBS.
+ *
+ * Results:
+ *      TRUE if PEBS is supported, FALSE otherwise.
+ *
+ * Side effects:
+ *      None
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+PerfCtr_PEBSAvailable(void)
+{
+   CPUIDRegs regs;
+   __GET_CPUID(0, &regs);
+   if (CPUID_IsVendorIntel(&regs) &&
+       (X86MSR_GetMSR(MSR_MISC_ENABLE) &
+        (MSR_MISC_ENABLE_EMON_AVAILABLE | MSR_MISC_ENABLE_PEBS_UNAVAILABLE))
+       == MSR_MISC_ENABLE_EMON_AVAILABLE) {
+      CPUIDRegs hvendor;
+
+      /*
+       * Hyper-V doesn't support PEBS and may #GP if we try to write the
+       * PEBS enable MSR so always consider PEBS un-available on Hyper-V -
+       * PR 1039970.
+       */
+      return !PerfCtr_HypervisorCPUIDSig(&hvendor) ||
+             !CPUID_IsRawVendor(&hvendor,
+                                CPUID_HYPERV_HYPERVISOR_VENDOR_STRING);
+   }
+   return FALSE;
+}
+
+/*
+ *----------------------------------------------------------------------
+ *
+ *  PerfCtr_PTAvailable --
+ *
+ *      Checks if this CPU is capable of PT(Intel processor trace).
+ *
+ * Results:
+ *      TRUE if PT is supported, FALSE otherwise.
+ *
+ * Side effects:
+ *      None
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+PerfCtr_PTAvailable(void)
+{
+   CPUIDRegs regs;
+   __GET_CPUID(0, &regs);
+   if (CPUID_IsVendorIntel(&regs)) {
+      __GET_CPUID2(7, 0, &regs);
+      return (regs.ebx & CPUID_INTERNAL_MASK_PT) != 0;
+   }
+   return FALSE;
+}
+
+#endif // _PERFCTR_ARCH_H

vmmon-only/include/pgtbl.h

@@ -0,0 +1,142 @@
+/*********************************************************
+ * Copyright (C) 2002,2014-2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef __PGTBL_H__
+#   define __PGTBL_H__
+
+
+#include <linux/highmem.h>
+
+#include "compat_pgtable.h"
+#include "compat_spinlock.h"
+#include "compat_page.h"
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * PgtblVa2MPNLocked --
+ *
+ *    Walks through the hardware page tables to try to find the pte
+ *    associated to a virtual address.  Then maps PTE to MPN.
+ *
+ * Results:
+ *    INVALID_MPN on failure
+ *    mpn         on success
+ *
+ * Side effects:
+ *    None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE MPN
+PgtblVa2MPNLocked(struct mm_struct *mm, // IN: Mm structure of a process
+                  VA addr)              // IN: Address in the virtual address
+                                        //     space of that process
+{
+   pgd_t *pgd;
+   compat_p4d_t *p4d;
+   MPN mpn;
+
+   pgd = pgd_offset(mm, addr);
+   if (pgd_present(*pgd) == 0) {
+      return INVALID_MPN;
+   }
+   if (pgd_large(*pgd)) {
+      /* Linux kernel does not support PGD huge pages. */
+      /* return pgd_pfn(*pgd) + ((addr & PGD_MASK) >> PAGE_SHIFT); */
+      return INVALID_MPN;
+   }
+
+   p4d = compat_p4d_offset(pgd, addr);
+   if (compat_p4d_present(*p4d) == 0) {
+      return INVALID_MPN;
+   }
+   if (compat_p4d_large(*p4d)) {
+      mpn = compat_p4d_pfn(*p4d) + ((addr & ~COMPAT_P4D_MASK) >> PAGE_SHIFT);
+   } else {
+      pud_t *pud;
+
+      pud = pud_offset(p4d, addr);
+      if (pud_present(*pud) == 0) {
+         return INVALID_MPN;
+      }
+      if (pud_large(*pud)) {
+         mpn = pud_pfn(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
+      } else {
+         pmd_t *pmd;
+
+         pmd = pmd_offset(pud, addr);
+         if (pmd_present(*pmd) == 0) {
+            return INVALID_MPN;
+         }
+         if (pmd_large(*pmd)) {
+            mpn = pmd_pfn(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
+         } else {
+            pte_t *pte;
+
+            pte = pte_offset_map(pmd, addr);
+            if (pte_present(*pte) == 0) {
+               pte_unmap(pte);
+               return INVALID_MPN;
+            }
+            mpn = pte_pfn(*pte);
+            pte_unmap(pte);
+         }
+      }
+   }
+   if (mpn >= INVALID_MPN) {
+      mpn = INVALID_MPN;
+   }
+   return mpn;
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * PgtblVa2MPN --
+ *
+ *    Walks through the hardware page tables of the current process to try to
+ *    find the page structure associated to a virtual address.
+ *
+ * Results:
+ *    Same as PgtblVa2MPNLocked()
+ *
+ * Side effects:
+ *    None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE MPN
+PgtblVa2MPN(VA addr)  // IN
+{
+   struct mm_struct *mm;
+   MPN mpn;
+
+   /* current->mm is NULL for kernel threads, so use active_mm. */
+   mm = current->active_mm;
+   spin_lock(&mm->page_table_lock);
+   mpn = PgtblVa2MPNLocked(mm, addr);
+   spin_unlock(&mm->page_table_lock);
+   return mpn;
+}
+
+#endif /* __PGTBL_H__ */

vmmon-only/include/pshare_ext.h

@@ -0,0 +1,73 @@
+/*********************************************************
+ * Copyright (C) 2001,2014 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * pshare_ext.h --
+ *
+ *      VMKernel/VMMon <-> VMM transparent page sharing info.
+ */
+
+#ifndef _PSHARE_EXT_H
+#define _PSHARE_EXT_H
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_MODULE
+#include "includeCheck.h"
+
+#include "pagelist.h"
+#include "vm_basic_types.h"
+#include "vm_assert.h"
+
+/*
+ * constants
+ */
+
+#define PSHARE_PAGELIST_MAX             (PAGELIST_MAX)
+#define PSHARE_P2M_BUFFER_MPNS_MAX      (32)
+#define PSHARE_P2M_BUFFER_MPNS_DEFAULT  (5)
+#define PSHARE_P2M_BUFFER_SLOTS_PER_MPN (PAGE_SIZE / sizeof(PShare_P2MUpdate))
+#define PSHARE_P2M_BUFFER_SLOTS_MIN     (2)
+
+#define PSHARE_POISON_MARKER            (CONST64U(0xAAAAAAAAAAAAAAAA))
+
+#define PSHARE_SALT_UNSET       0
+#define PSHARE_SALT_DEFAULT     1
+
+MY_ASSERTS(PSHARE_EXT,
+           ASSERT_ON_COMPILE(PSHARE_PAGELIST_MAX <= PAGELIST_MAX);)
+
+/*
+ * types
+ */
+
+typedef union {
+   MPN mpn;
+   uint64 vpmemRef;
+} PShare_P2MUpdateReference;
+
+#define PSHARE_SET_INVALID_P2MUPDATE_REFERENCE(ref) ((ref).vpmemRef = CONST64U(-1))
+#define PSHARE_IS_INVALID_P2MUPDATE_REFERENCE(ref) ((ref).vpmemRef == CONST64U(-1))
+
+typedef struct PShare_P2MUpdate {
+   BPN bpn;
+   PShare_P2MUpdateReference reference;
+} PShare_P2MUpdate;
+#endif

vmmon-only/include/ptsc.h

@@ -0,0 +1,234 @@
+/*********************************************************
+ * Copyright (C) 1998-2014,2017,2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * ptsc.h --
+ *
+ *      Pseudo TSC
+ */
+
+#ifndef _PTSC_H_
+#define _PTSC_H_
+
+#define INCLUDE_ALLOW_VMX
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#include "includeCheck.h"
+
+#include "rateconv.h"
+
+/*
+ * RDTSC and PTSC_Get.
+ *
+ * RDTSC reads the hardware timestamp counter on the current physical
+ * CPU.  In general, the TSC is *not* a globally consistent timer that
+ * runs at a constant rate.  Any code that still assumes it is should
+ * be corrected; see PR 20499.
+ *
+ * PTSC_Get returns a pseudo-TSC that runs at approximately the
+ * maximum speed of physical CPU 0's TSC and is approximately globally
+ * consistent.  It is available both at userlevel and in the monitor,
+ * with different implementations.  In the vmkernel, Timer_PseudoTSC
+ * provides similar functionality.
+ *
+ */
+
+/* TS stands for "timestamp", which is in units of "cycles" */
+typedef uint64 VmAbsoluteTS; // a particular point in time (in cycles)
+typedef int64  VmRelativeTS; // a signed delta in cycles
+typedef uint64 VmIntervalTS; // an unsigned delta in cycles
+typedef uint64 VmAbsoluteUS; // a particular point in time (in us)
+typedef int64  VmRelativeUS; // a signed delta in us
+typedef uint64 VmIntervalUS; // an unsigned delta in us
+
+/*
+ * Compare two VmAbsoluteTS's using comparison operator op, allowing
+ * for wrap.  The assumption is that differences should not be more
+ * than 2**63, so a larger difference is taken as negative.
+ */
+#define COMPARE_TS(ts1, op, ts2) (((int64) ((ts1) - (ts2))) op 0)
+
+#define MAX_ABSOLUTE_TS \
+   ((VmAbsoluteTS) CONST64U(0xffffffffffffffff))
+
+/*
+ * Largest possible unambiguous difference between two VmAbsoluteTS's
+ * according to COMPARE_TS's method of comparison.
+ */
+#define MAX_RELATIVE_TS \
+   ((VmRelativeTS) CONST64(0x7fffffffffffffff))
+
+#define MAX_ABSOLUTE_US \
+   ((VmAbsoluteUS) CONST64U(0xffffffffffffffff))
+
+typedef struct PTSCInfo {
+   Bool             hwTSCsSynced;
+   Bool             hwTSCsAdjusted;
+   uint8            _pad[6];
+   int64            hz;
+   uint32           kHz;
+   uint32           mHz;
+
+   RateConv_Ratio   usToCycles;
+   RateConv_Ratio   cyclesToUs;
+   RateConv_Ratio   msToCycles;
+   RateConv_Ratio   cyclesToNs;
+} PTSCInfo;
+
+extern PTSCInfo ptscInfo;
+
+Bool PTSC_Init(uint64 tscHz);
+VmAbsoluteTS PTSC_InitialCount(const char *module,
+                               const char *option,
+                               Bool stress,
+                               VmIntervalTS freq,
+                               VmAbsoluteTS defaultCnt);
+Bool PTSC_HasPerfectlySynchronizedTSCs(void);
+
+static INLINE int64
+PTSC_Hz(void)
+{
+   ASSERT(ptscInfo.hz);
+   return ptscInfo.hz;
+}
+
+static INLINE uint32
+PTSC_KHz(void)
+{
+   ASSERT(ptscInfo.kHz);
+   return ptscInfo.kHz;
+}
+
+static INLINE uint32
+PTSC_MHz(void)
+{
+   ASSERT(ptscInfo.mHz);
+   return ptscInfo.mHz;
+}
+
+#if defined(VM_X86_64) || defined(VM_ARM_64)
+
+/* 
+ * Conversions to/from cycles.  Note that the conversions operate on
+ * signed values, so be careful when taking the difference of two
+ * VmAbsoluteTS (which is unsigned) that that value is not out of range
+ * of the signed type.
+ */
+
+static INLINE VmRelativeTS
+PTSC_USToCycles(int64 us)
+{
+   return Muls64x32s64(us, ptscInfo.usToCycles.mult, ptscInfo.usToCycles.shift);
+}
+
+static INLINE VmRelativeTS
+PTSC_MSToCycles(int64 ms)
+{
+   return Muls64x32s64(ms, ptscInfo.msToCycles.mult, ptscInfo.msToCycles.shift);
+}
+
+static INLINE int64
+PTSC_CyclesToNS(VmRelativeTS ts)
+{
+   return Muls64x32s64(ts, ptscInfo.cyclesToNs.mult, ptscInfo.cyclesToNs.shift);
+}
+
+static INLINE int64
+PTSC_CyclesToUS(VmRelativeTS ts)
+{
+   return Muls64x32s64(ts, ptscInfo.cyclesToUs.mult, ptscInfo.cyclesToUs.shift);
+}
+
+#else
+
+/* 32-bit Muls64x32s64 too big to justify inlining. */
+VmRelativeTS PTSC_USToCycles(int64 us);
+VmRelativeTS PTSC_MSToCycles(int64 ms);
+int64 PTSC_CyclesToNS(VmRelativeTS ts);
+int64 PTSC_CyclesToUS(VmRelativeTS ts);
+
+#endif
+
+#if defined(VMX86_SERVER) && defined(VMX86_VMX)
+
+/*
+ * ESX with userworld VMX
+ */
+#include "user_layout.h"
+
+static INLINE VmAbsoluteTS
+PTSC_Get(void)
+{
+   extern __thread User_ThreadData vmkUserTdata;
+   VmAbsoluteTS ptsc;
+
+   if (vmkUserTdata.magic != USER_THREADDATA_MAGIC) {
+      return 0;
+   }
+   ptsc = vmkUserTdata.u.pseudoTSCGet(&vmkUserTdata);
+   ASSERT((int64)ptsc >= 0);
+   return ptsc;
+}
+
+#else
+
+/*
+ * Monitor and hosted VMX
+ */
+
+VmAbsoluteTS PTSC_Get(void);
+
+#endif
+
+/* 
+ *-----------------------------------------------------------------------------
+ *
+ * PTSC_HasSynchronizedTSCs --
+ *
+ *      Returns TRUE iff the platform TSCs are known to be synchronized.
+ * 
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE Bool
+PTSC_HasSynchronizedTSCs(void)
+{
+   return ptscInfo.hwTSCsSynced;
+}
+
+
+/* 
+ *-----------------------------------------------------------------------------
+ *
+ * PTSC_HostAdjustedTSCs --
+ *
+ *      Returns TRUE if the platform may have adjusted TSCs in an attempt
+ *      to sync them up.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE Bool
+PTSC_HostAdjustedTSCs(void)
+{
+   return ptscInfo.hwTSCsAdjusted;
+}
+
+#endif /* ifndef _PTSC_H_ */

vmmon-only/include/rateconv.h

@@ -0,0 +1,126 @@
+/*********************************************************
+ * Copyright (C) 2003-2016, 2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ *
+ * rateconv.h --
+ *
+ *     Parameters and functions for linear rate conversion of 64 bit
+ *     counters:
+ *
+ *       y = ((x * mult) >> shift) + add.
+ *
+ */
+
+#ifndef _VM_RATECONV_H_
+#define _VM_RATECONV_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+#include "vm_basic_types.h"
+#include "vm_basic_asm.h"
+#include "vm_assert.h"
+#include "vm_atomic.h"
+#include "versioned_atomic.h"
+
+#if defined __cplusplus
+extern "C" {
+#endif
+
+
+/* RateConv_Params is part of vmx<->vmmon interface (INIT_PSEUDO_TSC ioctl) */
+typedef struct RateConv_Params {
+   uint32 mult;  /* mult == 1 implies shift == 0. */
+   uint32 shift;
+   int64  add;
+} RateConv_Params;
+
+typedef struct RateConv_ParamsVolatile {
+   RateConv_Params p;
+   VersionedAtomic vers;
+} RateConv_ParamsVolatile;
+
+typedef struct RateConv_Ratio {
+   uint32 mult;
+   uint32 shift;
+} RateConv_Ratio;
+
+#define RATE_CONV_IDENTITY { 1, 0, 0 }  /* Out = in. */
+
+Bool RateConv_ComputeParams(uint64 inHz, uint64 inBase,
+		            uint64 outHz, uint64 outBase,
+		            RateConv_Params *conv);
+void RateConv_LogParams(const char *prefix,
+			uint64 inHz, uint64 inBase,
+			uint64 outHz, uint64 outBase,
+			const RateConv_Params *conv);
+Bool RateConv_ComputeRatio(uint64 inHz, uint64 outHz,
+                           RateConv_Ratio *ratio);
+void RateConv_LogRatio(const char *prefix,
+                       uint64 inHz, uint64 outHz,
+                       const RateConv_Ratio *ratio);
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * RateConv_Unsigned --
+ *
+ *      Apply rate conversion to an unsigned argument:
+ *       y = ((x * mult) >> shift) + add.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE uint64
+RateConv_Unsigned(const RateConv_Params *conv, uint64 x)
+{
+   return Mul64x3264(x, conv->mult, conv->shift) + (uint64)conv->add;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * RateConv_Signed --
+ *
+ *      Apply rate conversion to a signed argument:
+ *       y = ((x * mult) >> shift) + add.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE int64
+RateConv_Signed(const RateConv_Params *conv, int64 x)
+{
+   return Muls64x32s64(x, conv->mult, conv->shift) + conv->add;
+}
+
+
+#if defined __cplusplus
+} // extern "C"
+#endif
+
+#endif // _VM_RATECONV_H_

vmmon-only/include/sharedAreaType.h

@@ -0,0 +1,47 @@
+/*********************************************************
+ * Copyright (C) 2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * sharedAreaType.h --
+ *
+ *      This file contains shared area type definitions.
+ */
+
+#ifndef _SHAREDAREATYPE_H_
+#define _SHAREDAREATYPE_H_
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMX
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_MODULE
+
+#include "includeCheck.h"
+
+typedef enum {
+   SHARED_AREA_PER_VM_VMX = 0,
+   SHARED_AREA_INTER_VCPU_VMX,
+   SHARED_AREA_PER_VCPU_VMX,
+   SHARED_AREA_PER_VM,
+   SHARED_AREA_INTER_VCPU,
+   SHARED_AREA_PER_VCPU,
+   NUM_SHARED_AREAS
+} SharedAreaType;
+
+#endif // _SHAREDAREATYPE_H_

vmmon-only/include/sharedAreaVmmon.h

@@ -0,0 +1,62 @@
+/*********************************************************
+ * Copyright (C) 2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * sharedAreaVmmon.h --
+ *
+ *     VMMon shared area management.
+ */
+
+#ifndef SHARED_AREA_VMMON_H
+#define SHARED_AREA_VMMON_H
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+#include "iocontrols.h"
+
+struct VMDriver;
+
+/* A collection of pages backing a shared area region. */
+typedef struct SharedAreaVmmonRegion {
+   PageCnt pagesPerVcpu;
+   MPN *pages;
+} SharedAreaVmmonRegion;
+
+/* The VMMon Driver shared area. */
+typedef struct SharedAreaVmmon {
+   SharedAreaVmmonRegion regions[NUM_SHARED_AREAS];
+} SharedAreaVmmon;
+
+/* Request for a backing MPN for a shared area region at a given VCPU/offset. */
+typedef struct SharedAreaVmmonRequest {
+   SharedAreaType type;
+   Vcpuid vcpu;
+   PageCnt offset;
+} SharedAreaVmmonRequest;
+
+SharedAreaVmmon *SharedAreaVmmon_Init(struct VMDriver *vm);
+void SharedAreaVmmon_Cleanup(SharedAreaVmmon *area);
+Bool SharedAreaVmmon_RegisterRegion(struct VMDriver *driver,
+                                    VMSharedAreaRegistrationBlock *block);
+Bool SharedAreaVmmon_ValidateRegionArgs(struct VMDriver *driver,
+                                        VMSharedAreaRegistrationBlock *block);
+MPN SharedAreaVmmon_GetRegionMPN(struct VMDriver *vm,
+                                 SharedAreaVmmonRequest *request);
+#endif /* SHARED_AREA_VMMON_H */

vmmon-only/include/statVarsVmmon.h

@@ -0,0 +1,48 @@
+/*********************************************************
+ * Copyright (C) 2018-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * statVarsVmmon.h --
+ *
+ *     VMMon stat vars management.
+ */
+
+#ifndef STAT_VARS_VMMON_H
+#define STAT_VARS_VMMON_H
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+#include "iocontrols.h"
+
+struct VMDriver;
+
+/* The VMMon Driver stat vars area. */
+typedef struct StatVarsVmmon {
+   PageCnt pagesPerVcpu;
+   MPN *pages;
+} StatVarsVmmon;
+
+StatVarsVmmon *StatVarsVmmon_Init(struct VMDriver *vm);
+void StatVarsVmmon_Cleanup(StatVarsVmmon *statVars);
+Bool StatVarsVmmon_RegisterVCPU(struct VMDriver *driver,
+                                VMStatVarsRegistrationBlock *block);
+MPN StatVarsVmmon_GetRegionMPN(struct VMDriver *vm, Vcpuid vcpuid,
+                               PageCnt offset);
+#endif /* STAT_VARS_VMMON_H */

vmmon-only/include/uccostTable.h

@@ -0,0 +1,34 @@
+/*********************************************************
+ * Copyright (C) 1998-2014 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+UC(CALL_START)
+UC(BEGIN_BACK_TO_HOST)
+UC(SWITCHED_TO_MODULE)
+UC(VMX_SELECT_RETURN)
+UC(VMX_HANDLER_START)
+UC(VMX_SIGNAL)
+UC(MODULE_SIGNAL)
+UC(SWITCHING_TO_MONITOR)
+UC(DONE_BACK_TO_HOST)
+UC(CALL_END)
+
+#undef UC

vmmon-only/include/usercalldefs.h

@@ -0,0 +1,35 @@
+/*********************************************************
+ * Copyright (C) 2008,2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef USERCALLDEFS_H
+#define USERCALLDEFS_H
+
+#ifdef __linux__
+#include <linux/errno.h>
+#define USERCALL_RESTART (-ERESTARTNOINTR)
+#else
+#include <usercall.h>
+#define USERCALL_RESTART (USERCALL_NOP)
+#endif
+/*
+ * -1 to -4096 are reserved for syscall errors on Linux.  -1 is reserved for
+ * failing DeviceIoControl on Windows.
+ */
+#define USERCALL_VMX86ALLOCERR (-8192)
+
+#endif 

vmmon-only/include/vcpuid.h

@@ -0,0 +1,69 @@
+/*********************************************************
+ * Copyright (C) 1998-2014, 2016-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ *
+ * vcpuid.h --
+ *
+ *    Monitor's VCPU ID.
+ */
+
+#ifndef _VCPUID_H_
+#define _VCPUID_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#include "vm_basic_types.h"
+
+#if defined __cplusplus
+extern "C" {
+#endif
+
+typedef uint32 Vcpuid;                 // VCPU number
+
+#define VCPUID_INVALID  (~0U)
+
+#define BOOT_VCPU_ID     0
+#define IS_BOOT_VCPUID(vcpuid)  ((vcpuid) == BOOT_VCPU_ID)
+
+#define MAX_VCPUS            2048
+#define MAX_CORES_PER_SOCKET   64
+#define MAX_VCPU_SOCKETS      128
+
+/*
+ * There are several properties of the VM which change at the 128 VCPU
+ * boundary.  EFI firmware, x2APIC, and IOMMU are required among others.
+ */
+#define MAX_SMALL_VM_VCPUS 128
+
+/* Supported limit. */
+#define MAX_SUPPORTED_VCPUS   256
+
+#if defined __cplusplus
+} // extern "C"
+#endif
+
+#endif // ifndef _VCPUID_H_

vmmon-only/include/vcpuset.h

@@ -0,0 +1,896 @@
+/*********************************************************
+ * Copyright (C) 2002-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vcpuset.h --
+ *
+ *      ADT for a set of VCPUs.  Implemented as an array of bitmasks.
+ *
+ */
+
+#ifndef _VCPUSET_H_
+#define _VCPUSET_H_
+
+
+#define INCLUDE_ALLOW_VMX
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#include "vm_basic_asm.h"
+#include "vm_atomic.h"
+#include "vcpuid.h"
+#include "vcpuset_types.h"
+
+#if defined VMX86_VMX
+#   include "str.h"     /* Str_Snprintf */
+#   define VCS_SNPRINTF Str_Snprintf
+#elif defined MONITOR_APP
+#   include <stdio.h>   /* libc snprintf */
+#   if defined WIN32
+#      define VCS_SNPRINTF _snprintf
+#   else
+#      define VCS_SNPRINTF snprintf
+#   endif
+#elif defined VMM || defined VMKERNEL
+#   include "vm_libc.h" /* vmcore snprintf */
+#   define VCS_SNPRINTF snprintf
+#endif
+
+#ifdef VMX86_VMX
+#include "vmx.h"
+#endif
+
+
+/*
+ * A buffer for logging a VCPUSet must fit a maximally-populated set.  To
+ * balance brevity and readability, sets are formatted for printing like long
+ * hexadecimal numbers, with a '.' at every 64-VCPU subset boundary.  The
+ * highest-numbered VCPU in the set is printed first, followed by all slots
+ * for lower-numbered VCPUs, populated or not.  Leading zeroes are not printed.
+ *
+ * Examples, assuming a VCS_SUBSET_COUNT of 2:
+ * An empty set:                                            "0x0\0"
+ * A full set:              "0xffffffffffffffff.ffffffffffffffff\0"
+ * A set with only VCPU 50:                     "0x4000000000000\0"
+ * A set with only VCPU 80:            "0x10000.0000000000000000\0"
+ */
+#define VCS_BUF_SIZE (2 +                                         /* "0x"  */ \
+                      (VCS_SUBSET_COUNT * VCS_SUBSET_WIDTH / 4) + /* (hex) */ \
+                      (VCS_SUBSET_COUNT - 1) +                    /* '.'   */ \
+                      1)                                          /* NULL  */
+
+extern VCPUSet  vcpuSetFull;
+
+#define FOR_EACH_VCPU_IN_SET_WITH_MAX(_vcpuSet, _v, _numVcpus)                 \
+   do {                                                                        \
+      Vcpuid _v;                                                               \
+      const VCPUSet *__vcs = (_vcpuSet);                                       \
+      unsigned _subsetIdx = 0;                                                 \
+      uint64 _subset = VCPUSet_Subset(__vcs, _subsetIdx);                      \
+      unsigned _maxSubsets = VCS_VCPUID_SUBSET_IDX((_numVcpus) - 1) + 1;       \
+      ASSERT(_maxSubsets <= VCS_SUBSET_COUNT);                                 \
+      while ((_v = VCPUSet_FindFirstInSubset(__vcs, &_subset, &_subsetIdx,     \
+                                             _maxSubsets)) != VCPUID_INVALID) {\
+
+#define ROF_EACH_VCPU_IN_SET_WITH_MAX()                                 \
+      }                                                                 \
+   } while (0)
+
+
+#define FOR_EACH_VCPU_IN_SET(_vcpuSet, _v)                              \
+   FOR_EACH_VCPU_IN_SET_WITH_MAX(_vcpuSet, _v, MAX_VCPUS)
+
+#define ROF_EACH_VCPU_IN_SET()                                          \
+      }                                                                 \
+   } while (0)
+
+#define FOR_EACH_VCPU_IN_POPULATED_VCPUS(_vcpuSet, _v)                  \
+   FOR_EACH_VCPU_IN_SET_WITH_MAX(_vcpuSet, _v, NumVCPUs())
+
+#define ROF_EACH_VCPU_IN_POPULATED_VCPUS()                              \
+      }                                                                 \
+   } while (0)
+
+#define FOR_EACH_SUBSET_IN_SET(_setIndex)                                     \
+   do {                                                                       \
+      unsigned _setIndex;                                                     \
+      for (_setIndex = 0; _setIndex < VCS_SUBSET_COUNT; _setIndex++) {
+
+#define ROF_EACH_SUBSET_IN_SET()                                              \
+      }                                                                       \
+   } while (0)
+
+
+#define FOR_EACH_SUBSET_IN_SET_COUNTDOWN(_setIndex)                           \
+   do {                                                                       \
+      unsigned _setIndex = VCS_SUBSET_COUNT;                                  \
+      while (_setIndex-- > 0) {
+
+#define ROF_EACH_SUBSET_IN_SET_COUNTDOWN()                                    \
+      }                                                                       \
+   } while (0)
+
+
+#define FOR_EACH_POPULATED_SUBSET_IN_SET(_setIndex, _numVcpus)                \
+   do {                                                                       \
+      unsigned _setIndex;                                                     \
+      unsigned _maxSubsets = VCS_VCPUID_SUBSET_IDX(_numVcpus - 1);            \
+      for (_setIndex = 0; _setIndex <= _maxSubsets; _setIndex++) {
+
+#define ROF_EACH_POPULATED_SUBSET_IN_SET()                                    \
+      }                                                                       \
+   } while (0)
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Empty --
+ *
+ *      Clear all bits in a VCPUSet.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_Empty(VCPUSet *vcs)
+{
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      vcs->subset[idx] = 0;
+   } ROF_EACH_SUBSET_IN_SET();
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_IsEmpty --
+ *
+ *      Return TRUE iff a VCPUSet has no bits set.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+VCPUSet_IsEmpty(const VCPUSet *vcs)
+{
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      if (vcs->subset[idx] != 0) {
+         return FALSE;
+      }
+   } ROF_EACH_SUBSET_IN_SET();
+   return TRUE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Full --
+ *
+ *      Returns a pointer to a VCPUSet containing all valid VCPUs.
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE const VCPUSet *
+VCPUSet_Full(void)
+{
+   /*
+    * Read too early, we may get the wrong notion of how many
+    * vcpus the VM has. Cf. pr286243 and pr289186.
+    */
+#if defined (VMX86_VMX)
+   ASSERT(NumVCPUs() != 0 && !VCPUSet_IsEmpty(&vcpuSetFull));
+#endif
+   return &vcpuSetFull;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Copy --
+ *
+ *      Copy one VCPUSet's contents to another.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_Copy(VCPUSet *dest, const VCPUSet *src)
+{
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      dest->subset[idx] = src->subset[idx];
+   } ROF_EACH_SUBSET_IN_SET();
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Equals --
+ *
+ *      Compare two VCPUSets, return TRUE iff their contents match.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+VCPUSet_Equals(const VCPUSet *vcs1, const VCPUSet *vcs2)
+{
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      if (vcs1->subset[idx] != vcs2->subset[idx]) {
+         return FALSE;
+      }
+   } ROF_EACH_SUBSET_IN_SET();
+   return TRUE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_IsMember --
+ *
+ *      Return TRUE iff the given Vcpuid is present in a VCPUSet.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+VCPUSet_IsMember(const VCPUSet *vcs, Vcpuid v)
+{
+   ASSERT(v < MAX_VCPUS);
+   return (vcs->subset[VCS_VCPUID_SUBSET_IDX(v)] &
+           VCS_VCPUID_SUBSET_BIT(v)) != 0;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_AtomicIsMember --
+ *
+ *      Return TRUE iff the given Vcpuid is present in a VCPUSet.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+VCPUSet_AtomicIsMember(VCPUSet *vcs, Vcpuid v)
+{
+   volatile uint64 *subset = &vcs->subset[VCS_VCPUID_SUBSET_IDX(v)];
+   ASSERT(v < MAX_VCPUS);
+   return (Atomic_Read64(Atomic_VolatileToAtomic64(subset)) &
+           VCS_VCPUID_SUBSET_BIT(v)) != 0;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_FindFirst --
+ * VCPUSet_FindLast --
+ *
+ *      Find the first (lowest-numbered) or last (highest-numbered)
+ *      Vcpuid in a VCPUSet.
+ *
+ * Results:
+ *      Vcpuid if at least one is present in a set.
+ *      VCPUID_INVALID if the set is empty.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Vcpuid
+VCPUSet_FindFirst(const VCPUSet *vcs)
+{
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      uint64 subset = vcs->subset[idx];
+      if (subset != 0) {
+         return lssb64(subset) - 1 + (idx << VCS_SUBSET_SHIFT);
+      }
+   } ROF_EACH_SUBSET_IN_SET();
+   return VCPUID_INVALID;
+}
+
+static INLINE Vcpuid
+VCPUSet_FindLast(const VCPUSet *vcs)
+{
+   FOR_EACH_SUBSET_IN_SET_COUNTDOWN(idx) {
+      uint64 subset = vcs->subset[idx];
+      if (subset != 0) {
+         return mssb64(subset) - 1 + (idx << VCS_SUBSET_SHIFT);
+      }
+   } ROF_EACH_SUBSET_IN_SET_COUNTDOWN();
+   return VCPUID_INVALID;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_FindFirstInSubset --
+ *
+ *      Find the first (lowest-numbered) Vcpuid in the given subset of a
+ *      VCPUSet object.  It is required that *subset is initialized to
+ *      a subset of *vcs and *subsetIdx must hold the index of the subset
+ *      stored in *subset.
+ *
+ * Results:
+ *      Vcpuid if at least one is present in a set.
+ *      VCPUID_INVALID if the set is empty.
+ *
+ * Side effects:
+ *      This function is intended to be used for iterating over all bits
+ *      of a VCPUSet, it will modify *subset to clear the bit associated
+ *      with the returned Vcpuid (if any).  The *subsetIdx argument may
+ *      also be updated.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Vcpuid
+VCPUSet_FindFirstInSubset(const VCPUSet *vcs, uint64 *subset,
+                          unsigned *subsetIdx, unsigned maxSubsets)
+{
+   ASSERT(*subsetIdx < maxSubsets && maxSubsets <= VCS_SUBSET_COUNT);
+   do {
+      if (*subset != 0) {
+         unsigned bit = (unsigned)lssb64_0(*subset);
+         *subset &= ~(CONST64U(1) << bit);
+         return bit + (*subsetIdx << VCS_SUBSET_SHIFT);
+      }
+      ++*subsetIdx;
+      if (*subsetIdx < maxSubsets) {
+         *subset = vcs->subset[*subsetIdx];
+      }
+   } while (*subsetIdx < maxSubsets);
+   return VCPUID_INVALID;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Remove --
+ * VCPUSet_AtomicRemove --
+ *
+ *      Remove or atomically remove a single Vcpuid from a VCPUSet.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_Remove(VCPUSet *vcs, Vcpuid v)
+{
+   ASSERT(v < MAX_VCPUS);
+   vcs->subset[VCS_VCPUID_SUBSET_IDX(v)] &= ~VCS_VCPUID_SUBSET_BIT(v);
+}
+
+
+static INLINE void
+VCPUSet_AtomicRemove(VCPUSet *vcs, Vcpuid v)
+{
+   volatile uint64 *subset = &vcs->subset[VCS_VCPUID_SUBSET_IDX(v)];
+   ASSERT(v < MAX_VCPUS);
+   Atomic_And64(Atomic_VolatileToAtomic64(subset), ~VCS_VCPUID_SUBSET_BIT(v));
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_IncludeSet --
+ * VCPUSet_RemoveSet --
+ *
+ *      Add/remove all vcpus present in the set 'src' to/from the set 'dest'.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_IncludeSet(VCPUSet *dest, const VCPUSet *src)
+{
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      dest->subset[idx] |= src->subset[idx];
+   } ROF_EACH_SUBSET_IN_SET();
+}
+
+
+static INLINE void
+VCPUSet_RemoveSet(VCPUSet *dest, const VCPUSet *src)
+{
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      dest->subset[idx] &= ~src->subset[idx];
+   } ROF_EACH_SUBSET_IN_SET();
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Include --
+ * VCPUSet_AtomicInclude --
+ *
+ *      Add or atomically add a single Vcpuid to a VCPUSet.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_Include(VCPUSet *vcs, Vcpuid v)
+{
+   ASSERT(v < MAX_VCPUS);
+   vcs->subset[VCS_VCPUID_SUBSET_IDX(v)] |= VCS_VCPUID_SUBSET_BIT(v);
+}
+
+
+static INLINE void
+VCPUSet_AtomicInclude(VCPUSet *vcs, Vcpuid v)
+{
+   volatile uint64 *subset = &vcs->subset[VCS_VCPUID_SUBSET_IDX(v)];
+   ASSERT(v < MAX_VCPUS);
+   Atomic_Or64(Atomic_VolatileToAtomic64(subset), VCS_VCPUID_SUBSET_BIT(v));
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_AtomicTestInclude  --
+ *
+ *      Atomically add a single Vcpuid to a VCPUSet, and
+ *      return TRUE iff the given Vcpuid was present in the VCPUSet.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+VCPUSet_AtomicTestInclude(VCPUSet *vcs, Vcpuid v)
+{
+   volatile uint64 *subset = &vcs->subset[VCS_VCPUID_SUBSET_IDX(v)];
+   ASSERT(v < MAX_VCPUS);
+   return Atomic_TestSetBit64(Atomic_VolatileToAtomic64(subset),
+                              v & VCS_SUBSET_MASK);
+}
+
+
+#if defined(VMM) && !defined(MONITOR_APP)
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_PackCareful --
+ *
+ *      Pack a VCPUSet into the bytes at "ptr".
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_PackCareful(unsigned numVCPUs, const VCPUSet *vcs, void *ptr)
+{
+   memcpy(ptr, vcs->subset, (numVCPUs + 7) / 8);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_UnpackCareful --
+ *
+ *      Unpack a VCPUSet from the bytes at "ptr".
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_UnpackCareful(unsigned numVCPUs, VCPUSet *vcs, const void *ptr)
+{
+   memcpy(vcs->subset, ptr, (numVCPUs + 7) / 8);
+}
+#endif /* VMM && !MONITOR_APP */
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_PopulateRange --
+ *
+ *  Populates the given set with 'numVCPUs' VCPUs starting at 'firstVCPU'.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_PopulateRange(VCPUSet *vcs, unsigned firstVCPU,
+                          unsigned numVCPUs)
+{
+   unsigned sub;
+   unsigned lastVCPU      = firstVCPU + numVCPUs - 1;
+   unsigned firstSubset   = firstVCPU / VCS_SUBSET_WIDTH;
+   unsigned lastSubset    = lastVCPU / VCS_SUBSET_WIDTH;
+   unsigned lowMaskShift  = firstVCPU % VCS_SUBSET_WIDTH;
+   unsigned highMaskShift = VCS_SUBSET_WIDTH - 1 - lastVCPU % VCS_SUBSET_WIDTH;
+
+   ASSERT(firstSubset <= lastSubset && lastSubset < VCS_SUBSET_COUNT);
+
+   VCPUSet_Empty(vcs);
+   for (sub = firstSubset; sub <= lastSubset; sub++) {
+      vcs->subset[sub] = CONST64U(-1);
+   }
+   vcs->subset[firstSubset] &= (CONST64U(-1) << lowMaskShift);
+   vcs->subset[lastSubset] &= (CONST64U(-1) >> highMaskShift);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Populate --
+ *
+ *  Populates the given set with the VCPUs in [0, numVCPUs).
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_Populate(VCPUSet *vcs, unsigned numVCPUs)
+{
+   VCPUSet_PopulateRange(vcs, 0, numVCPUs);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Subset --
+ *
+ *      Return the specified subset of a VCPUSet.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE uint64
+VCPUSet_Subset(const VCPUSet *vcs,
+                   unsigned subset)
+{
+   ASSERT(subset < VCS_SUBSET_COUNT);
+   return vcs->subset[subset];
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_SubsetPtr --
+ *
+ *      Return a pointer to the specified subset of a VCPUSet.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE uint64 *
+VCPUSet_SubsetPtr(VCPUSet *vcs, unsigned subset)
+{
+   ASSERT(subset < VCS_SUBSET_COUNT);
+   return &vcs->subset[subset];
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_IsSupersetOrEqual --
+ *
+ *      Return TRUE iff vcs1 contains a superset of the VCPUs in vcs2
+ *      or vcs1 and vcs2 contain exactly the same VCPUs.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+VCPUSet_IsSupersetOrEqual(const VCPUSet *vcs1, const VCPUSet *vcs2)
+{
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      if (vcs2->subset[idx] & ~vcs1->subset[idx]) {
+         return FALSE;
+      }
+   } ROF_EACH_SUBSET_IN_SET();
+   return TRUE;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_IsSubsetOrEqual --
+ *
+ *      Return TRUE iff vcs1 contains a subset of the VCPUs in vcs2
+ *      or vcs1 and vcs2 contain exactly the same VCPUs.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Bool
+VCPUSet_IsSubsetOrEqual(const VCPUSet *vcs1, const VCPUSet *vcs2)
+{
+   return VCPUSet_IsSupersetOrEqual(vcs2, vcs1);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_MakeSingleton --
+ *
+ *      Add a single Vcpuid to a VCPUSet and remove all others.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_MakeSingleton(VCPUSet *vcs, Vcpuid v)
+{
+   VCPUSet_Empty(vcs);
+   VCPUSet_Include(vcs, v);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_FindSingleton --
+ *
+ *      Return the VCPU in the set iff vcs contains exactly one VCPU.
+ *      Return VCPUID_INVALID otherwise.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE Vcpuid
+VCPUSet_FindSingleton(const VCPUSet *vcs)
+{
+   uint64 foundSub = 0;
+   uint32 foundIdx = 0;
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      uint64 sub = vcs->subset[idx];
+      if (sub != 0) {
+         if (foundSub != 0 || (sub & (sub - 1)) != 0) {
+            return VCPUID_INVALID;
+         }
+         foundSub = sub;
+         foundIdx = idx;
+      }
+   } ROF_EACH_SUBSET_IN_SET();
+   if (foundSub != 0) {
+      return lssb64(foundSub) - 1 + (foundIdx << VCS_SUBSET_SHIFT);
+   } else {
+      return VCPUID_INVALID;
+   }
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_IsFull --
+ *
+ *  Returns true iff vcs contains the set of all vcpus.
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE Bool
+VCPUSet_IsFull(const VCPUSet *vcs)
+{
+   return VCPUSet_Equals(vcs, VCPUSet_Full());
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_AtomicReadWriteSubset --
+ *
+ *      For the nth aligned 64-VCPU subset of a VCPU set, atomically
+ *      read then write.  Return the contents read.  Set 0 is VCPUs
+ *      0-63 and set 1 is VCPUs 64-127.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE uint64
+VCPUSet_AtomicReadWriteSubset(VCPUSet *vcs, uint64 vcpus,
+                                  unsigned n)
+{
+   ASSERT(n < VCS_SUBSET_COUNT);
+   return Atomic_ReadWrite64(Atomic_VolatileToAtomic64(&vcs->subset[n]),
+                             vcpus);
+}
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Size --
+ *
+ *    Return the number of VCPUs in this set.
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE int
+VCPUSet_Size(const VCPUSet *vcs)
+{
+   int n = 0;
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      uint64 bits = vcs->subset[idx];
+      while (bits != 0) {
+         bits = bits & (bits - 1);
+         n++;
+      }
+   } ROF_EACH_SUBSET_IN_SET();
+   return n;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_UnionSubset --
+ *
+ *      Given an 64-bit value and a subset number, add the VCPUs
+ *      represented to the set.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_UnionSubset(VCPUSet *vcs, uint64 vcpus, unsigned n)
+{
+   ASSERT(n < VCS_SUBSET_COUNT);
+   vcs->subset[n] |= vcpus;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_SubtractSubset --
+ *
+ *      Given an 64-bit value and a subset number, remove the VCPUs
+ *      represented in the subset from the set.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_SubtractSubset(VCPUSet *vcs, uint64 vcpus, unsigned n)
+{
+   ASSERT(n < VCS_SUBSET_COUNT);
+   vcs->subset[n] &= ~vcpus;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_AtomicUnionSubset --
+ *
+ *      Given a 64-bit value and a subset number, atomically add
+ *      the VCPUs represented to the set.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_AtomicUnionSubset(VCPUSet *vcs, uint64 vcpus, unsigned n)
+{
+   uint64 *subsetPtr = &vcs->subset[n];
+   ASSERT(n < VCS_SUBSET_COUNT);
+   Atomic_Or64(Atomic_VolatileToAtomic64(subsetPtr), vcpus);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Invert --
+ *
+ *      Makes all non-present valid VCPUs in a set present and all
+ *      VCPUs present non-present.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_Invert(VCPUSet *vcs)
+{
+   VCPUSet temp;
+   VCPUSet_Copy(&temp, VCPUSet_Full());
+   VCPUSet_RemoveSet(&temp, vcs);
+   VCPUSet_Copy(vcs, &temp);
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_Intersection
+ *
+ *      Given two VCPUSets, populate the destination set with only the
+ *      VCPUs common to both.
+ *
+ *----------------------------------------------------------------------
+ */
+
+static INLINE void
+VCPUSet_Intersection(VCPUSet *dest, const VCPUSet *src)
+{
+   FOR_EACH_SUBSET_IN_SET(idx) {
+      dest->subset[idx] &= src->subset[idx];
+   } ROF_EACH_SUBSET_IN_SET();
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * VCPUSet_LogFormat --
+ *
+ *      Given a buffer of at least VCS_BUF_SIZE to fill, write into it a
+ *      string suitable for use in Log() or LOG().
+ *      Returns the buffer which was passed as an argument, after
+ *      writing the string.
+ *
+ *----------------------------------------------------------------------
+ */
+
+#ifdef VCS_SNPRINTF
+static INLINE char *
+VCPUSet_LogFormat(char *buf, size_t size, const VCPUSet *vcs)
+{
+   unsigned offset  = 0;
+   Vcpuid   highest = VCPUSet_FindLast(vcs);
+   unsigned idx     = (highest == VCPUID_INVALID) ? 0 : highest / 8;
+
+   ASSERT(size >= VCS_BUF_SIZE);
+#define VCS_LOGF(...)                                                   \
+   {                                                                    \
+      int ret = VCS_SNPRINTF(buf + offset, size - offset, __VA_ARGS__); \
+      ASSERT(0 <= ret && size >= offset && ret < (int)(size - offset)); \
+      offset += (unsigned)ret;                                          \
+   }
+   /* Print the leading value with no zero-extension. */
+   VCS_LOGF("%#x", ((unsigned char *)vcs)[idx]);
+
+   while (idx-- > 0) {
+      if ((idx + 1) % (VCS_SUBSET_WIDTH / 8) == 0) {
+         VCS_LOGF(".");
+      }
+      VCS_LOGF("%02x", ((unsigned char *)vcs)[idx]);
+   }
+   return buf;
+}
+#undef VCS_LOGF
+#endif
+
+
+#endif /* _VCPUSET_H_ */

vmmon-only/include/vcpuset_types.h

@@ -0,0 +1,71 @@
+/*********************************************************
+ * Copyright (C) 2002-2013, 2016-2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vcpuset_types.h --
+ *
+ *    ADT for a set of VCPUs.  Implemented as an array of bitmasks.
+ *
+ */
+
+#ifndef _VCPUSET_TYPES_H_
+#define _VCPUSET_TYPES_H_
+
+
+#define INCLUDE_ALLOW_VMX
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#include "vcpuid.h"
+#include "vm_assert.h"
+
+#define VCS_SUBSET_WIDTH                                                   64
+#define VCS_SUBSET_SHIFT                                                    6
+#define VCS_SUBSET_MASK               ((CONST64U(1) << VCS_SUBSET_SHIFT) - 1)
+#define VCS_SUBSET_COUNT                                                   32
+
+
+#define VCS_VCPUID_SUBSET_IDX(v)                    ((v) >> VCS_SUBSET_SHIFT)
+#define VCS_VCPUID_SUBSET_BIT(v)     (CONST64U(1) << ((v) & VCS_SUBSET_MASK))
+
+/*
+ * If you update this type, you also need to update the SEND_IPI line in
+ * bora/public/iocontrolsMacosTable.h.
+ */
+typedef struct VCPUSet {
+   uint64 subset[VCS_SUBSET_COUNT];
+} VCPUSet;
+
+MY_ASSERTS(VCPUSET_ASSERTS,
+           ASSERT_ON_COMPILE(VCS_SUBSET_WIDTH * VCS_SUBSET_COUNT >= MAX_VCPUS);
+           /*
+            * Catch changes in VCPUSet which need to be reflected in
+            * bora/public/iocontrolsMacosTable.h.
+            */
+           ASSERT_ON_COMPILE(VCS_SUBSET_COUNT == 32);
+           /*
+            * There is code that depends on sizeof(VCPUSet) being a power of
+            * 2 in at least vcpuHotPlug.c and possible other places.
+            */
+            ASSERT_ON_COMPILE((sizeof(VCPUSet) & (sizeof(VCPUSet) - 1)) == 0);
+)
+#endif

vmmon-only/include/versioned_atomic.h

@@ -0,0 +1,274 @@
+/*********************************************************
+ * Copyright (C) 1998,2015-2016 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * versioned_atomic.h --
+ *
+ *      This file implements versioned atomic synchronization, which allows
+ *      concurrent accesses to shared data by a single writer and multiple
+ *      readers. The algorithm is lock-free (it does not use a mutual exclusion
+ *      mechanism), and the writer is wait-free (it never waits for readers).
+ *
+ *      Many-writer/many-reader can be implemented on top of versioned atomics
+ *      by adding an external mutual exclusion mechanism to serialize all
+ *      writer threads into a single logical writer thread. This is preferable
+ *      for cases where readers are expected to greatly outnumber writers.
+ *
+ *      Implementation
+ *      --------------
+ *      This implementation is based on Leslie Lamport's paper "Concurrent
+ *      Reading and Writing", Communications of the ACM, November 1977
+ *      (http://url.eng.vmware.com/udcc ).
+ *
+ *      Lamport's algorithm was designed for systems in which the reader and
+ *      the writers run on separate computers and where the version variables
+ *      (v0 and v1 in this implementation) are stored on a shared hard disk. So
+ *      it did not assume that the version variables could be accessed
+ *      atomically. Instead, it assumed that the version variables were made of
+ *      multiple basic units of data, called "digits", and that each digit
+ *      could be accessed atomically. The upper-left corner of page 4 of the
+ *      paper precisely and concisely describes the algorithm, and in
+ *      particular the order in which digits of the version variables must be
+ *      accessed (left to right, right to left, or no particular order).
+ *
+ *      This implementation is designed for systems in which the reader and the
+ *      writers run on the same computer and where the version variables are
+ *      stored in shared memory. So it assumes that the version variables can
+ *      be accessed atomically. As a result, this implementation uses a
+ *      simplified version of Lamport's algorithm, where:
+ *      o Each left to right or right to left access in Lamport's algorithm is
+ *        replaced with a single atomic access in this implementation.
+ *      o Each access in no particular order in Lamport's algorithm is replaced
+ *        with a single non-atomic access in this implementation.
+ *      Note: The writer is the only thread which writes to version variables,
+ *            so it is allowed to read them non-atomically.
+ *
+ *      Multiple concurrent writers to the version variables are not allowed.
+ *      Even if writers are working on lock-free or disjoint data, the version
+ *      variables are not interlocked for read-modify-write. See PR514764.
+ *
+ *      Recursive use of versioned atomics in writers is currently not
+ *      supported. See PR514764.
+ */
+
+#ifndef _VERSIONED_ATOMIC_H
+#define _VERSIONED_ATOMIC_H
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+#include "vm_basic_asm.h"
+#include "vm_assert.h"
+#include "vm_atomic.h"
+
+#if defined __cplusplus
+extern "C" {
+#endif
+
+
+/*
+ * Users with attribute(packed) structs must ensure any
+ * VersionedAtomic members are marked as ALIGNED(4).  Unfortunately
+ * the compiler cannot be trusted to align our substruct correctly
+ * (PR515329).  If the enclosing struct is packed, the VersionedAtomic
+ * alignment requested below will be ignored!
+ */
+
+typedef struct {
+   uint32 v0;
+   uint32 v1;
+} ALIGNED(4) VersionedAtomic;
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * VersionedAtomic_BeginWrite --
+ *
+ *      Called by the writer to indicate that it is about to write to the
+ *      shared data protected by 'versions'. Effectively locks out all readers
+ *      until VersionedAtomic_EndWrite() is called.
+ *
+ * Results:
+ *      None
+ *
+ * Side effects:
+ *      None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE void
+VersionedAtomic_BeginWrite(VersionedAtomic *versions) // IN
+{
+   /* Catch improper initialization or unsupported recursive use. */
+   ASSERT(versions->v1 == versions->v0);
+
+   /*
+    * Do not use Atomic_Inc32() below: it is more expensive than
+    * Atomic_Write32() and unnecessary: only the write needs to be atomic, not
+    * the whole read + inc + write.
+    */
+#if !defined VMM
+   /*
+    * Atomic_Write32() below requires the address to be naturally aligned, but
+    * currently only checks that condition when VMM is defined (tracked by
+    * PR1055836). So until that PR is fixed, we must check that condition here
+    * when VMM is not defined.
+    */
+   ASSERT(((uintptr_t)&versions->v0 % sizeof versions->v0) == 0);
+#endif
+   Atomic_Write32((Atomic_uint32 *)&versions->v0, versions->v0 + 1);
+
+   /*
+    * The write to 'versions->v0' must be observed by all other OS CPUs before
+    * the write(s) to the shared data.
+    */
+   SMP_W_BARRIER_W();
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * VersionedAtomic_EndWrite --
+ *
+ *      Called by the writer to indicate that it has finished writing to the
+ *      shared data protected by 'versions'. Lets pending and new readers
+ *      proceed.
+ *
+ * Results:
+ *      None
+ *
+ * Side effects:
+ *      None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE void
+VersionedAtomic_EndWrite(VersionedAtomic *versions) // IN
+{
+   /*
+    * The write(s) to the shared data must be observed by all other OS CPUs
+    * before the write to 'versions->v1'.
+    */
+   SMP_W_BARRIER_W();
+
+   ASSERT(versions->v1 + 1 == versions->v0);
+
+#if !defined VMM
+   /*
+    * Atomic_Write32() below requires the address to be naturally aligned, but
+    * currently only checks that condition when VMM is defined (tracked by
+    * PR1055836). So until that PR is fixed, we must check that condition here
+    * when VMM is not defined.
+    */
+   ASSERT(((uintptr_t)&versions->v1 % sizeof versions->v1) == 0);
+#endif
+   Atomic_Write32((Atomic_uint32 *)&versions->v1, versions->v0);
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * VersionedAtomic_BeginTryRead --
+ *
+ *      Called by a reader before it starts reading the shared data protected
+ *      by 'versions'.
+ *
+ * Results:
+ *      A version number for the caller to pass to the matching call to
+ *      VersionedAtomic_EndTryRead(), if any.
+ *
+ * Side effects:
+ *      None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE uint32
+VersionedAtomic_BeginTryRead(VersionedAtomic const *versions) // IN
+{
+   uint32 readVersion = Atomic_Read32((Atomic_uint32 *)&versions->v1);
+
+   /*
+    * The read from 'versions->v1' must happen on this CPU before the read(s)
+    * from the shared data.
+    */
+   SMP_R_BARRIER_R();
+
+   return readVersion;
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * VersionedAtomic_EndTryRead --
+ *
+ *      Called by a reader after it has finished reading the shared data
+ *      protected by 'versions', to confirm that the data is consistent, i.e.
+ *      that the writer has not modified the data while the reader was reading
+ *      it.
+ *
+ * Results:
+ *      TRUE if the shared data read between VersionedAtomic_BeginTryRead() and
+ *      this call is consistent.
+ *      FALSE otherwise.
+ *
+ * Side effects:
+ *      None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE Bool
+VersionedAtomic_EndTryRead(VersionedAtomic const *versions, // IN
+                           uint32 readVersion)              // IN
+{
+   /*
+    * The read(s) from the shared data must happen on this CPU before the read
+    * from 'versions->v0'.
+    */
+   SMP_R_BARRIER_R();
+
+   /*
+    * There is a theoretical ABA issue here: if the writer updates the shared
+    * data exactly 2^32 times while a reader reads it, the read will
+    * incorrectly be considered consistent. In practice, this issue is
+    * unlikely, so we ignore it. But should we need it, we could make the issue
+    * even less likely by using 64-bit version variables.
+    */
+   return LIKELY(   Atomic_Read32((Atomic_uint32 *)&versions->v0)
+                 == readVersion);
+}
+
+
+#if defined __cplusplus
+} // extern "C"
+#endif
+
+#endif //_VERSIONED_ATOMIC_H

vmmon-only/include/vm_asm.h

@@ -0,0 +1,88 @@
+/*********************************************************
+ * Copyright (C) 1998-2014,2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vm_asm.h
+ *
+ *	asm macros
+ */
+
+#ifndef _VM_ASM_H_
+#define _VM_ASM_H_
+
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_USERLEVEL
+#include "includeCheck.h"
+
+#ifdef VM_ARM_64
+#include "vm_asm_arm64.h"
+#else
+
+#include "vm_basic_asm.h"
+#include "x86msr.h"
+#include "vm_asm_x86.h"
+
+
+static INLINE void
+SET_FS64(uint64 fs64)
+{
+   X86MSR_SetMSR(MSR_FSBASE, fs64);
+}
+
+
+static INLINE void
+SET_GS64(uint64 gs64)
+{
+   X86MSR_SetMSR(MSR_GSBASE, gs64);
+}
+
+static INLINE void
+SET_KernelGS64(uint64 kgs64)
+{
+   X86MSR_SetMSR(MSR_KERNELGSBASE, kgs64);
+}
+
+
+static INLINE uint64
+GET_FS64(void)
+{
+   return X86MSR_GetMSR(MSR_FSBASE);
+}
+
+
+static INLINE uint64
+GET_GS64(void)
+{
+   return X86MSR_GetMSR(MSR_GSBASE);
+}
+
+
+static INLINE uint64
+GET_KernelGS64(void)
+{
+   return X86MSR_GetMSR(MSR_KERNELGSBASE);
+}
+
+#endif // VM_ARM_64
+#endif

vmmon-only/include/vm_asm_x86.h

@@ -0,0 +1,827 @@
+/*********************************************************
+ * Copyright (C) 1998-2014,2016-2017,2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vm_asm_x86.h
+ *
+ *      IA32 asm macros
+ */
+
+#ifndef _VM_ASM_X86_H_
+#define _VM_ASM_X86_H_
+
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_USERLEVEL
+#include "includeCheck.h"
+
+#include "x86types.h"
+#include "x86desc.h"
+#include "x86sel.h"
+#include "x86_basic_defs.h"
+#include "x86msr.h"
+
+#ifdef VM_X86_64
+#define _GETSET_DTR_TYPE DTR64
+#else
+#define _GETSET_DTR_TYPE DTR32
+#endif
+
+#ifdef __GNUC__
+
+/* ASSERT_ON_COMPILE_SELECTOR_SIZE:
+ *
+ *  - Selector must be 16-bits.
+ *  - If a constant is used, it better be only 16-bits.
+ *  - If it's not a constant, it must be Selector-sized. or less.
+ *
+ * Although aesthetically the following looks nicer, gcc is unable
+ * to produce a constant expression for it:
+ *
+ *  ASSERT_ON_COMPILE(sizeof(Selector) == 2 &&                            \
+ *                    ((__builtin_constant_p(expr) ? ((expr) >> 16) == 0) \
+ *                                                 : sizeof(expr) <= 2)
+ */
+#ifndef USE_UBSAN
+#define ASSERT_ON_COMPILE_SELECTOR_SIZE(expr)                                \
+   ASSERT_ON_COMPILE(sizeof(Selector) == 2 &&                                \
+                     ((__builtin_constant_p(expr) && ((expr) >> 16) == 0) || \
+                      sizeof(expr) <= 2))
+#else
+#define ASSERT_ON_COMPILE_SELECTOR_SIZE(expr)
+#endif
+
+
+/* Checked against the Intel manual and GCC --hpreg */
+static INLINE void
+_Set_GDT(_GETSET_DTR_TYPE *dtr)
+{
+   __asm__(
+      "lgdt %0"
+      :
+      : "m" (*dtr)
+   );
+}
+
+/* Checked against the Intel manual and GCC --hpreg */
+static INLINE void
+_Set_IDT(_GETSET_DTR_TYPE *dtr)
+{
+   __asm__(
+      "lidt %0"
+      :
+      : "m" (*dtr)
+   );
+}
+
+/*
+ * Checked against the Intel manual and GCC --hpreg
+ * volatile because there's a hidden input (the [IG]DTR) that can change
+ * without the compiler knowing it.
+ */
+static INLINE void
+_Get_GDT(_GETSET_DTR_TYPE *dtr)
+{
+   __asm__ __volatile__(
+      "sgdt %0"
+      : "=m" (*dtr)
+   );
+}
+
+/*
+ * Checked against the Intel manual and GCC --hpreg
+ * volatile because the [IG]DT can change without the compiler knowing it
+ * (when we use l[ig]dt).
+ */
+static INLINE void
+_Get_IDT(_GETSET_DTR_TYPE *dtr)
+{
+   __asm__ __volatile__(
+      "sidt %0"
+      : "=m" (*dtr)
+   );
+}
+
+
+#define SET_LDT(expr)                                                   \
+   do {                                                                 \
+      const Selector _set_ldt_sel = (Selector)(expr);                   \
+      ASSERT_ON_COMPILE_SELECTOR_SIZE(expr);                            \
+      /* lldt reads from the GDT; don't sink any writes. */             \
+      COMPILER_MEM_BARRIER();                                           \
+      /* Checked against the Intel manual and GCC --hpreg */            \
+      __asm__("lldt %0"                                                 \
+              :                                                         \
+              : "rm" (_set_ldt_sel));                                   \
+   } while (0)
+
+
+/* Checked against the Intel manual and GCC --hpreg
+ *
+ * volatile because the LDT can change without the compiler knowing it
+ * (when we use lldt).
+ */
+static INLINE void
+_GET_LDT(Selector * const result)
+{
+   __asm__ __volatile__("sldt %0"
+                        : "=rm" (*result));
+}
+
+
+#define GET_LDT(var)    \
+   do {                 \
+      _GET_LDT(&(var)); \
+   } while (0)
+
+
+#define _BUILD_SET_R(func, reg)        \
+   static INLINE void                  \
+   func(uintptr_t r)                   \
+   {                                   \
+      __asm__("mov %0, %%" #reg        \
+              : /* no outputs */       \
+              :  "r" (r)               \
+              : "memory");             \
+   }
+
+/*
+ * The inline asm is marked 'volatile' because CRs and DRs can change
+ * without the compiler knowing it (when there is a page fault, when a
+ * breakpoint occurs, and moreover it seems there is no way to teach
+ * gcc that smsw clobbers cr0 for example).
+ */
+#define _BUILD_GET_R(func, reg)                         \
+   static INLINE uintptr_t                              \
+   func(void)                                           \
+   {                                                    \
+      uintptr_t result;                                 \
+      __asm__ __volatile__("mov %%" #reg ", %0"         \
+                           : "=r" (result));            \
+      return result;                                    \
+   }
+
+_BUILD_SET_R(_SET_CR0, cr0)
+_BUILD_SET_R(_SET_CR2, cr2)
+_BUILD_SET_R(_SET_CR3, cr3)
+_BUILD_SET_R(_SET_CR4, cr4)
+_BUILD_SET_R(_SET_CR8, cr8)
+
+_BUILD_GET_R(_GET_CR0, cr0)
+_BUILD_GET_R(_GET_CR2, cr2)
+_BUILD_GET_R(_GET_CR3, cr3)
+_BUILD_GET_R(_GET_CR4, cr4)
+_BUILD_GET_R(_GET_CR8, cr8)
+
+_BUILD_SET_R(_SET_DR0, dr0)
+_BUILD_SET_R(_SET_DR1, dr1)
+_BUILD_SET_R(_SET_DR2, dr2)
+_BUILD_SET_R(_SET_DR3, dr3)
+_BUILD_SET_R(_SET_DR6, dr6)
+_BUILD_SET_R(_SET_DR7, dr7)
+
+_BUILD_GET_R(_GET_DR0, dr0)
+_BUILD_GET_R(_GET_DR1, dr1)
+_BUILD_GET_R(_GET_DR2, dr2)
+_BUILD_GET_R(_GET_DR3, dr3)
+_BUILD_GET_R(_GET_DR6, dr6)
+_BUILD_GET_R(_GET_DR7, dr7)
+
+#define SET_CR2(expr) SET_CR_DR(CR, 2, expr)
+
+#define SET_SEGREG(reg, expr)                                          \
+   do {                                                                \
+      const Selector _set_segreg_sel = (Selector)(expr);               \
+      ASSERT_ON_COMPILE_SELECTOR_SIZE(expr);                           \
+      /* mov to Sreg reads from the [GL]DT; don't sink any writes. */  \
+      COMPILER_MEM_BARRIER();                                          \
+      /* Checked against the Intel manual and GCC --hpreg */           \
+      __asm__("movw %0, %%" #reg                                       \
+              :                                                        \
+              : "rm" (_set_segreg_sel));                               \
+   } while (0)
+
+#define SET_DS(expr) SET_SEGREG(ds, expr)
+#define SET_ES(expr) SET_SEGREG(es, expr)
+#define SET_FS(expr) SET_SEGREG(fs, expr)
+#define SET_GS(expr) SET_SEGREG(gs, expr)
+#define SET_SS(expr) SET_SEGREG(ss, expr)
+
+/* Checked against the Intel manual and GCC --hpreg
+ *
+ * volatile because the content of CS can change without the compiler
+ * knowing it (when we use call gates).
+ *
+ * XXX: The segment register getter functions have not been updated to
+ *      have stricter type checking like many other functions in this
+ *      file because they return a value, rather than taking an
+ *      argument.  Perhaps sometime in the future, a willing soul will
+ *      change these accessors to take an argument and at the same
+ *      time install better type checking.
+ */
+#define _BUILD_GET_SEG(func, reg)                \
+   static INLINE Selector                        \
+   func(void)                                    \
+   {                                             \
+      Selector result;                           \
+      __asm__ __volatile__("movw %%" #reg ", %0" \
+                           : "=rm" (result));    \
+      return result;                             \
+   }
+
+_BUILD_GET_SEG(GET_CS, cs)
+_BUILD_GET_SEG(GET_DS, ds)
+_BUILD_GET_SEG(GET_ES, es)
+_BUILD_GET_SEG(GET_FS, fs)
+_BUILD_GET_SEG(GET_GS, gs)
+_BUILD_GET_SEG(GET_SS, ss)
+
+
+#define SET_TR(expr)                                                    \
+   do {                                                                 \
+      const Selector _set_tr_sel = (Selector)(expr);                    \
+      ASSERT_ON_COMPILE_SELECTOR_SIZE(expr);                            \
+      /* ltr reads from the GDT; don't sink any writes. */              \
+      COMPILER_MEM_BARRIER();                                           \
+      /* Checked against the Intel manual and GCC --hpreg */            \
+      __asm__ __volatile__("ltr %0"                                     \
+                           :                                            \
+                           : "rm" (_set_tr_sel) : "memory");            \
+   } while (0)
+
+/* Checked against the Intel manual and GCC --hpreg
+
+   volatile because the content of TR can change without the compiler knowing
+   it (when we use task gates). */
+static INLINE void
+_GET_TR(Selector * const result)
+{
+   __asm__ __volatile__("str %0"
+                        : "=rm" (*result));
+}
+
+#define GET_TR(var)    \
+   do {                \
+      _GET_TR(&(var)); \
+   } while (0)
+
+
+/* Checked against the Intel manual and GCC --hpreg
+
+   We use this to restore interrupts, so this cannot be reordered around
+   by gcc */
+static INLINE void
+_Set_flags(uintptr_t f)
+{
+   __asm__ __volatile__(
+      "push %0" "\n\t"
+      "popf"
+      :
+      : "g" (f)
+      : "memory", "cc"
+   );
+}
+
+
+
+/* Checked against the Intel manual and GCC --hpreg
+
+   volatile because gcc 2.7.2.3 doesn't know when eflags are modified (it
+   seems to ignore the "cc" clobber). gcc 2.95.2 is ok: it optimize 2
+   successive calls only if there is no instructions in between. */
+static INLINE uintptr_t
+_Get_flags(void)
+{
+   uintptr_t result;
+
+   __asm__ __volatile__(
+      "pushf"  "\n\t"
+      "pop %0"
+      : "=rm" (result)
+      :
+      : "memory"
+   );
+
+   return result;
+}
+
+#define SAVE_FLAGS(var) do { \
+   var = _Get_flags();       \
+} while (0)
+
+static INLINE Bool
+HwInterruptsEnabled(uint32 eflags)
+{
+   return (eflags & EFLAGS_IF) != 0;
+}
+
+/* Checked against the Intel manual and GCC --hpreg */
+static INLINE void
+CLTS(void)
+{
+   __asm__ __volatile__ ("clts");
+}
+
+
+/* Beginning of the section whose correctness has NOT been checked */
+#define FNCLEX()         __asm__("fnclex" ::);
+
+/* TLB_INVALIDATE_PAGE is not checked yet */
+#define TLB_INVALIDATE_PAGE(_addr) do { \
+     __asm__ __volatile__("invlpg %0": :"m" (*(char *) (_addr)):"memory"); \
+} while (0)
+
+#define RESTORE_FLAGS _Set_flags
+#define ENABLE_INTERRUPTS() __asm__ __volatile__ ("sti": : :"memory")
+#define CLEAR_INTERRUPTS()  __asm__ __volatile__ ("cli": : :"memory")
+#define RAISE_INTERRUPT(_x)  __asm__ __volatile__("int %0" :: "g" (_x))
+#define RETURN_FROM_INT()   __asm__ __volatile__("iret" :: )
+
+/* End of the section whose correctness has NOT been checked */
+
+#elif defined _MSC_VER  /* !__GNUC__ */
+
+#define _BUILD_SET_DR(func, reg)                      \
+   static INLINE void                                 \
+   func(uintptr_t r)                                  \
+   {                                                  \
+      __writedr(reg, r);                              \
+   }
+
+#define _BUILD_GET_DR(func, reg)                      \
+   static INLINE uintptr_t                            \
+   func(void)                                         \
+   {                                                  \
+      return __readdr(reg);                           \
+   }
+
+#define _BUILD_SET_CR(func, reg)                      \
+   static INLINE void                                 \
+   func(uintptr_t r)                                  \
+   {                                                  \
+      __writecr##reg(r);                              \
+   }
+
+#define _BUILD_GET_CR(func, reg)                      \
+   static INLINE uintptr_t                            \
+   func(void)                                         \
+   {                                                  \
+      return __readcr##reg();                         \
+   }
+
+_BUILD_SET_DR(_SET_DR0, 0)
+_BUILD_SET_DR(_SET_DR1, 1)
+_BUILD_SET_DR(_SET_DR2, 2)
+_BUILD_SET_DR(_SET_DR3, 3)
+_BUILD_SET_DR(_SET_DR6, 6)
+_BUILD_SET_DR(_SET_DR7, 7)
+
+_BUILD_GET_DR(_GET_DR0, 0)
+_BUILD_GET_DR(_GET_DR1, 1)
+_BUILD_GET_DR(_GET_DR2, 2)
+_BUILD_GET_DR(_GET_DR3, 3)
+_BUILD_GET_DR(_GET_DR6, 6)
+_BUILD_GET_DR(_GET_DR7, 7)
+
+_BUILD_SET_CR(_SET_CR0, 0);
+_BUILD_SET_CR(_SET_CR3, 3);
+_BUILD_SET_CR(_SET_CR4, 4);
+_BUILD_SET_CR(_SET_CR8, 8);
+
+_BUILD_GET_CR(_GET_CR0, 0);
+_BUILD_GET_CR(_GET_CR2, 2);
+_BUILD_GET_CR(_GET_CR3, 3);
+_BUILD_GET_CR(_GET_CR4, 4);
+_BUILD_GET_CR(_GET_CR8, 8);
+
+static INLINE void
+_Set_GDT(_GETSET_DTR_TYPE *dtr)
+{
+#ifdef VM_X86_64
+   _lgdt(dtr);
+#else
+   /*
+    * timetools 32-bit targets use an old version of MSVC that doesn't support
+    * the _lgdt intrinsic. Luckily, 32-bit still supports inline asm.
+    */
+   __asm mov eax, dtr
+   __asm lgdt [eax]
+#endif
+}
+
+static INLINE void
+_Get_GDT(_GETSET_DTR_TYPE *dtr)
+{
+#ifdef VM_X86_64
+   _sgdt(dtr);
+#else
+   /*
+    * timetools 32-bit targets use an old version of MSVC that doesn't support
+    * the _sgdt intrinsic. Luckily, 32-bit still supports inline asm.
+    */
+   __asm mov eax, dtr
+   __asm sgdt [eax]
+#endif
+}
+
+static INLINE void
+_Set_IDT(_GETSET_DTR_TYPE *dtr)
+{
+   __lidt(dtr);
+}
+
+static INLINE void
+_Get_IDT(_GETSET_DTR_TYPE *dtr)
+{
+   __sidt(dtr);
+}
+
+
+#define ENABLE_INTERRUPTS() _enable()
+#define CLEAR_INTERRUPTS()  _disable()
+
+#define SAVE_FLAGS(x) do { \
+   (x) = __readeflags();   \
+} while (0)
+
+#define RESTORE_FLAGS(x) __writeeflags(x)
+
+
+#ifdef VM_X86_32
+
+#define SET_TR(_tr)      _Set_TR(_tr)
+#define SET_LDT(_tr)     _Set_LDT(_tr)
+
+#define GET_TR(_tr)      do { _tr = _Get_TR();  } while (0)
+#define GET_LDT(_tr)     do { _tr = _Get_LDT(); } while (0)
+
+#define SET_CR2(_reg)    __asm mov eax, _reg __asm mov cr2, eax
+#define CLTS()           __asm clts
+
+#define FNCLEX()         __asm fnclex
+
+#define TLB_INVALIDATE_PAGE(_addr) {  \
+   void *_a = (_addr); \
+   __asm mov eax, _a __asm invlpg [eax] \
+}
+
+#define RAISE_INTERRUPT(_x)  {__asm int _x }
+#define RETURN_FROM_INT()   {__asm iretd }
+
+
+static INLINE void
+SET_DS(Selector val)
+{
+   __asm mov ax, val
+   __asm mov ds, ax
+}
+
+static INLINE void
+SET_ES(Selector val)
+{
+   __asm mov ax, val
+   __asm mov es, ax
+}
+
+static INLINE void
+SET_FS(Selector val)
+{
+   __asm mov ax, val
+   __asm mov fs, ax
+}
+
+static INLINE void
+SET_GS(Selector val)
+{
+   __asm mov ax, val
+   __asm mov gs, ax
+}
+
+static INLINE void
+SET_SS(Selector val)
+{
+   __asm mov ax, val
+   __asm mov ss, ax
+}
+
+static INLINE Selector
+GET_FS(void)
+{
+   Selector _v;
+   __asm mov _v,fs
+   return _v;
+}
+
+static INLINE Selector
+GET_GS(void)
+{
+   Selector _v;
+   __asm mov _v,gs
+   return _v;
+}
+
+static INLINE Selector
+GET_DS(void)
+{
+   Selector _v;
+   __asm mov _v,ds
+   return _v;
+}
+
+static INLINE Selector
+GET_ES(void)
+{
+   Selector _v;
+   __asm mov _v,es
+   return _v;
+}
+
+static INLINE Selector
+GET_SS(void)
+{
+   Selector _v;
+   __asm mov _v,ss
+   return _v;
+}
+
+static INLINE Selector
+GET_CS(void)
+{
+   Selector _v;
+   __asm mov _v,cs
+   return _v;
+}
+
+static INLINE void
+_Set_LDT(Selector val)
+{
+   __asm lldt val
+}
+
+static INLINE void
+_Set_TR(Selector val)
+{
+   __asm ltr val
+}
+
+static INLINE Selector
+_Get_LDT(void)
+{
+   Selector sel;
+   __asm sldt sel
+   return sel;
+}
+
+static INLINE Selector
+_Get_TR(void)
+{
+   Selector sel;
+   __asm str sel
+   return sel;
+}
+
+#endif /* !VM_X86_32 */
+#endif /* !__GNUC__ && !_MSC_VER */
+
+
+#define SET_CR_DR(regType, regNum, expr)                     \
+   do {                                                      \
+      /* Ensure no implicit truncation of 'expr' */          \
+      ASSERT_ON_COMPILE(sizeof(expr) <= sizeof(uintptr_t));  \
+      _SET_##regType##regNum(expr);                          \
+   } while (0)
+
+#define GET_CR_DR(regType, regNum, var) \
+   do {                                 \
+      var = _GET_##regType##regNum();   \
+   } while (0)
+
+#define SET_DR0(expr) SET_CR_DR(DR, 0, expr)
+#define SET_DR1(expr) SET_CR_DR(DR, 1, expr)
+#define SET_DR2(expr) SET_CR_DR(DR, 2, expr)
+#define SET_DR3(expr) SET_CR_DR(DR, 3, expr)
+#define SET_DR6(expr) SET_CR_DR(DR, 6, expr)
+#define SET_DR7(expr) SET_CR_DR(DR, 7, expr)
+
+#define GET_DR0(var)  GET_CR_DR(DR, 0, var)
+#define GET_DR1(var)  GET_CR_DR(DR, 1, var)
+#define GET_DR2(var)  GET_CR_DR(DR, 2, var)
+#define GET_DR3(var)  GET_CR_DR(DR, 3, var)
+#define GET_DR6(var)  GET_CR_DR(DR, 6, var)
+#define GET_DR7(var)  GET_CR_DR(DR, 7, var)
+
+/* Undefine GET_CR0; it is defined in mach_asm.h for SLES cross-compile */
+#undef GET_CR0
+#define GET_CR0(var)  GET_CR_DR(CR, 0, var)
+#define GET_CR2(var)  GET_CR_DR(CR, 2, var)
+#define GET_CR3(var)  GET_CR_DR(CR, 3, var)
+#define GET_CR4(var)  GET_CR_DR(CR, 4, var)
+#define GET_CR8(var)  GET_CR_DR(CR, 8, var)
+
+#define SET_CR0(expr) SET_CR_DR(CR, 0, expr)
+#define SET_CR3(expr) SET_CR_DR(CR, 3, expr)
+#define SET_CR4(expr) SET_CR_DR(CR, 4, expr)
+#define SET_CR8(expr) SET_CR_DR(CR, 8, expr)
+
+/*
+ * When the ULM macro-defines INTERRUPTS_ENABLED, attempting to define
+ * it as a function produces hard-to-diagnose compile-time errors.
+ */
+#if !defined(ULM) && !defined(INTERRUPTS_ENABLED)
+static INLINE Bool
+INTERRUPTS_ENABLED(void)
+{
+   uintptr_t flags;
+   SAVE_FLAGS(flags);
+   return ((flags & EFLAGS_IF) != 0);
+}
+#endif
+
+/*
+ * [GS]ET_[GI]DT() are defined as macros wrapping a function
+ * so we can pass the argument implicitly by reference (requires
+ * a macro) and get type checking too (requires a function).
+ */
+
+#define SET_GDT(_gdt)    _Set_GDT(&(_gdt))
+#define GET_GDT(_gdt)    _Get_GDT(&(_gdt))
+
+#define SET_IDT(_idt)    _Set_IDT(&(_idt))
+#define GET_IDT(_idt)    _Get_IDT(&(_idt))
+
+#if !defined(VMKERNEL)
+#define NO_INTERRUPTS_BEGIN() do { \
+                                   uintptr_t _flags; \
+                                   SAVE_FLAGS(_flags); \
+                                   CLEAR_INTERRUPTS();
+
+#define NO_INTERRUPTS_END()        RESTORE_FLAGS(_flags); \
+                                 } while(0)
+#endif
+
+
+#if defined (__GNUC__) || (defined (_MSC_VER) && defined (VM_X86_32))
+static INLINE unsigned
+CURRENT_CPL(void)
+{
+   return SELECTOR_RPL(GET_CS());
+}
+#endif
+
+
+static INLINE uint64
+RDPMC(int counter)
+{
+#ifdef __GNUC__
+#ifdef VM_X86_64
+   uint64 pmcLow;
+   uint64 pmcHigh;
+
+   __asm__ __volatile__(
+      "rdpmc"
+      : "=a" (pmcLow), "=d" (pmcHigh)
+      : "c" (counter)
+   );
+
+   return pmcHigh << 32 | pmcLow;
+#else
+   uint64 pmc;
+
+   __asm__ __volatile__(
+      "rdpmc"
+      : "=A" (pmc)
+      : "c" (counter)
+   );
+
+   return pmc;
+#endif
+#elif defined _MSC_VER
+   return __readpmc(counter);
+#endif
+}
+
+
+#if defined(VMM) || defined(VMKERNEL) || defined(FROBOS) || defined (ULM)
+static INLINE uint64
+__XGETBV(int cx)
+{
+#ifdef __GNUC__
+#ifdef VM_X86_64
+   uint64 lowval, hival;
+   __asm__ __volatile__(
+      "xgetbv"
+      : "=a" (lowval), "=d" (hival)
+      : "c" (cx)
+   );
+   return hival << 32 | lowval;
+#else
+   uint64 val;
+   __asm__ __volatile__(
+      "xgetbv"
+      : "=A" (val)
+      : "c" (cx)
+   );
+   return val;
+#endif
+#elif defined _MSC_VER
+   return _xgetbv((unsigned)cx);
+#endif
+}
+
+static INLINE void
+__XSETBV(int cx, uint64 val)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__(
+      "xsetbv"
+      : /* no outputs */
+      : "a" ((uint32)val), "d" ((uint32)(val >> 32)), "c" (cx)
+    );
+#elif defined _MSC_VER
+   _xsetbv((unsigned)cx, val);
+#endif
+}
+
+static INLINE uint64
+GET_XCR0(void)
+{
+   return __XGETBV(0);
+}
+
+#define SET_XCR0(val) __XSETBV(0, val)
+
+static INLINE void
+SET_XCR0_IF_NEEDED(uint64 newVal, uint64 oldVal)
+{
+#ifndef VMM_BOOTSTRAP
+   ASSERT(oldVal == GET_XCR0());
+   if (newVal != oldVal) {
+      SET_XCR0(newVal);
+   }
+#endif
+}
+#endif
+
+
+static INLINE uint32
+RDTSCP_AuxOnly(void)
+{
+#ifdef __GNUC__
+   uint32 tscLow, tscHigh, tscAux;
+
+   __asm__ __volatile__(
+      "rdtscp"
+      : "=a" (tscLow), "=d" (tscHigh), "=c" (tscAux)
+   );
+
+   return tscAux;
+#elif defined _MSC_VER
+   uint32 tscAux;
+
+   __rdtscp(&tscAux);
+
+   return tscAux;
+#endif
+}
+
+
+static INLINE uint64
+RDTSCP(void)
+{
+#ifdef __GNUC__
+   uint32 tscLow, tscHigh, tscAux;
+
+   __asm__ __volatile__(
+      "rdtscp"
+      :"=a" (tscLow), "=d" (tscHigh), "=c" (tscAux)
+   );
+
+   return QWORD(tscHigh, tscLow);
+#elif defined _MSC_VER
+   uint32 tscAux;
+
+   return __rdtscp(&tscAux);
+#endif
+}
+
+#endif

vmmon-only/include/vm_assert.h

@@ -0,0 +1,333 @@
+/*********************************************************
+ * Copyright (C) 1998-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vm_assert.h --
+ *
+ *	The basic assertion facility for all VMware code.
+ *
+ *      For proper use, see bora/doc/assert and
+ *      http://vmweb.vmware.com/~mts/WebSite/guide/programming/asserts.html.
+ */
+
+#ifndef _VM_ASSERT_H_
+#define _VM_ASSERT_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMKDRIVERS
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+// XXX not necessary except some places include vm_assert.h improperly
+#include "vm_basic_types.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/*
+ * Some bits of vmcore are used in VMKernel code and cannot have
+ * the VMKERNEL define due to other header dependencies.
+ */
+#if defined(VMKERNEL) && !defined(VMKPANIC)
+#define VMKPANIC 1
+#endif
+
+/*
+ * Internal macros, functions, and strings
+ *
+ * The monitor wants to save space at call sites, so it has specialized
+ * functions for each situation.  User level wants to save on implementation
+ * so it uses generic functions.
+ */
+
+#if !defined VMM || defined MONITOR_APP // {
+
+#if defined (VMKPANIC) 
+#include "vmk_assert.h"
+#else /* !VMKPANIC */
+#include <linux/kernel.h>
+#define _ASSERT_PANIC(name) \
+           Panic(_##name##Fmt "\n", __FILE__, __LINE__)
+#define _ASSERT_PANIC_BUG(bug, name) \
+           Panic(_##name##Fmt " bugNr=%d\n", __FILE__, __LINE__, bug)
+#define _ASSERT_PANIC_NORETURN(name) \
+           Panic(_##name##Fmt "\n", __FILE__, __LINE__)
+#define _ASSERT_PANIC_BUG_NORETURN(bug, name) \
+           Panic(_##name##Fmt " bugNr=%d\n", __FILE__, __LINE__, bug)
+#endif /* VMKPANIC */
+
+#endif // }
+
+
+// These strings don't have newline so that a bug can be tacked on.
+#define _AssertPanicFmt            "PANIC %s:%d"
+#define _AssertAssertFmt           "ASSERT %s:%d"
+#define _AssertVerifyFmt           "VERIFY %s:%d"
+#define _AssertNotImplementedFmt   "NOT_IMPLEMENTED %s:%d"
+#define _AssertNotReachedFmt       "NOT_REACHED %s:%d"
+#define _AssertMemAllocFmt         "MEM_ALLOC %s:%d"
+#define _AssertNotTestedFmt        "NOT_TESTED %s:%d"
+
+
+/*
+ * Panic and log functions
+ */
+
+void Log(const char *fmt, ...) PRINTF_DECL(1, 2);
+void Warning(const char *fmt, ...) PRINTF_DECL(1, 2);
+#if defined VMKPANIC
+void Panic_SaveRegs(void);
+
+NORETURN void Panic_NoSave(const char *fmt, ...) PRINTF_DECL(1, 2);
+
+#define Panic(fmt...) do { \
+   Panic_SaveRegs();       \
+   Panic_NoSave(fmt);      \
+} while(0)
+
+#else
+#define Panic panic
+#endif
+
+void LogThrottled(uint32 *count, const char *fmt, ...) PRINTF_DECL(2, 3);
+void WarningThrottled(uint32 *count, const char *fmt, ...) PRINTF_DECL(2, 3);
+
+
+#ifndef ASSERT_IFNOT
+   /*
+    * PR 271512: When compiling with gcc, catch assignments inside an ASSERT.
+    *
+    * 'UNLIKELY' is defined with __builtin_expect, which does not warn when
+    * passed an assignment (gcc bug 36050). To get around this, we put 'cond'
+    * in an 'if' statement and make sure it never gets executed by putting
+    * that inside of 'if (0)'. We use gcc's statement expression syntax to
+    * make ASSERT an expression because some code uses it that way.
+    *
+    * Since statement expression syntax is a gcc extension and since it's
+    * not clear if this is a problem with other compilers, the ASSERT
+    * definition was not changed for them. Using a bare 'cond' with the
+    * ternary operator may provide a solution.
+    */
+
+   #ifdef __GNUC__
+      #define ASSERT_IFNOT(cond, panic)                                       \
+         ({if (UNLIKELY(!(cond))) { panic; if (0) { if (cond) {;}}} (void)0;})
+   #else
+      #define ASSERT_IFNOT(cond, panic)                                       \
+         (UNLIKELY(!(cond)) ? (panic) : (void)0)
+   #endif
+#endif
+
+
+/*
+ * Assert, panic, and log macros
+ *
+ * Some of these are redefined below undef !VMX86_DEBUG.
+ * ASSERT() is special cased because of interaction with Windows DDK.
+ */
+
+#if defined VMX86_DEBUG
+#undef  ASSERT
+#define ASSERT(cond) ASSERT_IFNOT(cond, _ASSERT_PANIC(AssertAssert))
+#define ASSERT_BUG(bug, cond) \
+           ASSERT_IFNOT(cond, _ASSERT_PANIC_BUG(bug, AssertAssert))
+#endif
+
+#undef  VERIFY
+#define VERIFY(cond) \
+           ASSERT_IFNOT(cond, _ASSERT_PANIC_NORETURN(AssertVerify))
+#define VERIFY_BUG(bug, cond) \
+           ASSERT_IFNOT(cond, _ASSERT_PANIC_BUG_NORETURN(bug, AssertVerify))
+
+#define PANIC()        _ASSERT_PANIC(AssertPanic)
+#define PANIC_BUG(bug) _ASSERT_PANIC_BUG(bug, AssertPanic)
+
+#define ASSERT_NOT_IMPLEMENTED(cond) \
+           ASSERT_IFNOT(cond, NOT_IMPLEMENTED())
+
+#if defined VMKPANIC || defined VMM
+#define NOT_IMPLEMENTED()        _ASSERT_PANIC_NORETURN(AssertNotImplemented)
+#else
+#define NOT_IMPLEMENTED()        _ASSERT_PANIC(AssertNotImplemented)
+#endif
+
+#if defined VMM
+#define NOT_IMPLEMENTED_BUG(bug) \
+          _ASSERT_PANIC_BUG_NORETURN(bug, AssertNotImplemented)
+#else 
+#define NOT_IMPLEMENTED_BUG(bug) _ASSERT_PANIC_BUG(bug, AssertNotImplemented)
+#endif
+
+#if defined VMKPANIC || defined VMM
+#define NOT_REACHED()            _ASSERT_PANIC_NORETURN(AssertNotReached)
+#else
+#define NOT_REACHED()            _ASSERT_PANIC(AssertNotReached)
+#endif
+
+#define ASSERT_MEM_ALLOC(cond) \
+           ASSERT_IFNOT(cond, _ASSERT_PANIC(AssertMemAlloc))
+
+#ifdef VMX86_DEVEL
+#define NOT_TESTED()       Warning(_AssertNotTestedFmt "\n", __FILE__, __LINE__)
+#else
+#define NOT_TESTED()       Log(_AssertNotTestedFmt "\n", __FILE__, __LINE__)
+#endif
+
+#define ASSERT_NO_INTERRUPTS()  ASSERT(!INTERRUPTS_ENABLED())
+#define ASSERT_HAS_INTERRUPTS() ASSERT(INTERRUPTS_ENABLED())
+
+#define ASSERT_NOT_TESTED(cond) (UNLIKELY(!(cond)) ? NOT_TESTED() : (void)0)
+#define NOT_TESTED_ONCE()       DO_ONCE(NOT_TESTED())
+
+#define NOT_TESTED_1024()                                               \
+   do {                                                                 \
+      static MONITOR_ONLY(PERVCPU) uint16 count = 0;                    \
+      if (UNLIKELY(count == 0)) { NOT_TESTED(); }                       \
+      count = (count + 1) & 1023;                                       \
+   } while (0)
+
+#define LOG_ONCE(...) DO_ONCE(Log(__VA_ARGS__))
+
+
+/*
+ * Redefine macros that are only in debug versions
+ */
+
+#if !defined VMX86_DEBUG // {
+
+#undef  ASSERT
+#define ASSERT(cond)          ((void)0)
+#define ASSERT_BUG(bug, cond) ((void)0)
+
+/*
+ * Expand NOT_REACHED() as appropriate for each situation.
+ *
+ * Mainly, we want the compiler to infer the same control-flow
+ * information as it would from Panic().  Otherwise, different
+ * compilation options will lead to different control-flow-derived
+ * errors, causing some make targets to fail while others succeed.
+ *
+ * VC++ has the __assume() built-in function which we don't trust
+ * (see bug 43485); gcc has no such construct; we just panic in
+ * userlevel code.  The monitor doesn't want to pay the size penalty
+ * (measured at 212 bytes for the release vmm for a minimal infinite
+ * loop; panic would cost even more) so it does without and lives
+ * with the inconsistency.
+ */
+
+#if defined VMKPANIC || defined VMM
+#undef  NOT_REACHED
+#if defined __GNUC__ && (__GNUC__ > 4 || __GNUC__ == 4 && __GNUC_MINOR__ >= 5)
+#define NOT_REACHED() (__builtin_unreachable())
+#else
+#define NOT_REACHED() ((void)0)
+#endif
+#else
+// keep debug definition
+#endif
+
+#undef LOG_UNEXPECTED
+#define LOG_UNEXPECTED(bug)     ((void)0)
+
+#undef  ASSERT_NOT_TESTED
+#define ASSERT_NOT_TESTED(cond) ((void)0)
+#undef  NOT_TESTED
+#define NOT_TESTED()            ((void)0)
+#undef  NOT_TESTED_ONCE
+#define NOT_TESTED_ONCE()       ((void)0)
+#undef  NOT_TESTED_1024
+#define NOT_TESTED_1024()       ((void)0)
+
+#endif // !VMX86_DEBUG }
+
+
+/*
+ * Compile-time assertions.
+ *
+ * ASSERT_ON_COMPILE does not use the common
+ * switch (0) { case 0: case (e): ; } trick because some compilers (e.g. MSVC)
+ * generate code for it.
+ *
+ * The implementation uses both enum and typedef because the typedef alone is
+ * insufficient; gcc allows arrays to be declared with non-constant expressions
+ * (even in typedefs, where it makes no sense).
+ *
+ * NOTE: if GCC ever changes so that it ignores unused types altogether, this
+ * assert might not fire!  We explicitly mark it as unused because GCC 4.8+
+ * uses -Wunused-local-typedefs as part of -Wall, which means the typedef will
+ * generate a warning.
+ */
+
+#if defined(_Static_assert) || defined(__cplusplus) ||                         \
+    !defined(__GNUC__) || __GNUC__ < 4 || (__GNUC__ == 4 && __GNUC_MINOR__ < 6)
+#define ASSERT_ON_COMPILE(e) \
+   do { \
+      enum { AssertOnCompileMisused = ((e) ? 1 : -1) }; \
+      UNUSED_TYPE(typedef char AssertOnCompileFailed[AssertOnCompileMisused]); \
+   } while (0)
+#else
+#define ASSERT_ON_COMPILE(e) \
+   do {                      \
+      _Static_assert(e, #e); \
+   } while (0)
+#endif
+
+/*
+ * To put an ASSERT_ON_COMPILE() outside a function, wrap it
+ * in MY_ASSERTS().  The first parameter must be unique in
+ * each .c file where it appears.  For example,
+ *
+ * MY_ASSERTS(FS3_INT,
+ *    ASSERT_ON_COMPILE(sizeof(FS3_DiskLock) == 128);
+ *    ASSERT_ON_COMPILE(sizeof(FS3_DiskLockReserved) == DISK_BLOCK_SIZE);
+ *    ASSERT_ON_COMPILE(sizeof(FS3_DiskBlock) == DISK_BLOCK_SIZE);
+ *    ASSERT_ON_COMPILE(sizeof(Hardware_DMIUUID) == 16);
+ * )
+ *
+ * Caution: ASSERT() within MY_ASSERTS() is silently ignored.
+ * The same goes for anything else not evaluated at compile time.
+ */
+
+#define MY_ASSERTS(name, assertions) \
+   static INLINE void name(void) {   \
+      assertions                     \
+   }
+
+/*
+ * Avoid generating extra code due to asserts which are required by
+ * Clang static analyzer, e.g. right before a statement would fail, using
+ * the __clang_analyzer__ macro defined only when clang SA is parsing files.
+ */
+#ifdef __clang_analyzer__
+#define ANALYZER_ASSERT(cond) ASSERT(cond)
+#else
+#define ANALYZER_ASSERT(cond) ((void)0)
+#endif
+
+#ifdef __cplusplus
+} /* extern "C" */
+#endif
+
+#endif /* ifndef _VM_ASSERT_H_ */

vmmon-only/include/vm_basic_asm_x86.h

@@ -0,0 +1,543 @@
+/*********************************************************
+ * Copyright (C) 1998-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vm_basic_asm_x86.h
+ *
+ *	Basic IA32 asm macros
+ */
+
+#ifndef _VM_BASIC_ASM_X86_H_
+#define _VM_BASIC_ASM_X86_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#if defined __cplusplus
+extern "C" {
+#endif
+
+
+#ifdef VM_X86_64
+/*
+ * The gcc inline asm uses the "A" constraint which differs in 32 & 64
+ * bit mode.  32 bit means eax and edx, 64 means rax or rdx.
+ */
+#error "x86-64 not supported"
+#endif
+
+/*
+ * XTEST
+ *     Return TRUE if processor is in transaction region.
+ *
+ *  Using condition codes as output values (=@ccnz) requires gcc6 or
+ *  above.  Clang does not support condition codes as output
+ *  constraints.
+ *
+ */
+#if defined(__GNUC__) && (defined(VMM) || defined(VMKERNEL) || defined(FROBOS))
+static INLINE Bool
+xtest(void)
+{
+   Bool result;
+#if defined(__clang__)
+   __asm__ __volatile__("xtest\n"
+                        "setnz %%al"
+                        : "=a" (result) : : "cc");
+#else
+   __asm__ __volatile__("xtest"
+                        : "=@ccnz" (result) : : "cc");
+#endif
+   return result;
+}
+
+#endif /* __GNUC__ */
+
+
+/*
+ * FXSAVE/FXRSTOR
+ *     save/restore SIMD/MMX fpu state
+ *
+ * The pointer passed in must be 16-byte aligned.
+ *
+ * Intel and AMD processors behave differently w.r.t. fxsave/fxrstor. Intel
+ * processors unconditionally save the exception pointer state (instruction
+ * ptr., data ptr., and error instruction opcode). FXSAVE_ES1 and FXRSTOR_ES1
+ * work correctly for Intel processors.
+ *
+ * AMD processors only save the exception pointer state if ES=1. This leads to a
+ * security hole whereby one process/VM can inspect the state of another process
+ * VM. The AMD recommended workaround involves clobbering the exception pointer
+ * state unconditionally, and this is implemented in FXRSTOR_AMD_ES0. Note that
+ * FXSAVE_ES1 will only save the exception pointer state for AMD processors if
+ * ES=1.
+ *
+ * The workaround (FXRSTOR_AMD_ES0) only costs 1 cycle more than just doing an
+ * fxrstor, on both AMD Opteron and Intel Core CPUs.
+ */
+#if defined(__GNUC__)
+static INLINE void 
+FXSAVE_ES1(void *save)
+{
+   __asm__ __volatile__ ("fxsave %0\n" : "=m" (*(uint8 *)save) : : "memory");
+}
+
+static INLINE void 
+FXRSTOR_ES1(const void *load)
+{
+   __asm__ __volatile__ ("fxrstor %0\n"
+                         : : "m" (*(const uint8 *)load) : "memory");
+}
+
+static INLINE void 
+FXRSTOR_AMD_ES0(const void *load)
+{
+   uint64 dummy = 0;
+
+   __asm__ __volatile__ 
+       ("fnstsw  %%ax    \n"     // Grab x87 ES bit
+        "bt      $7,%%ax \n"     // Test ES bit
+        "jnc     1f      \n"     // Jump if ES=0
+        "fnclex          \n"     // ES=1. Clear it so fild doesn't trap
+        "1:              \n"
+        "ffree   %%st(7) \n"     // Clear tag bit - avoid poss. stack overflow
+        "fildl   %0      \n"     // Dummy Load from "safe address" changes all
+                                 // x87 exception pointers.
+        "fxrstor %1      \n"
+        :  
+        : "m" (dummy), "m" (*(const uint8 *)load)
+        : "ax", "memory");
+}
+#endif /* __GNUC__ */
+
+/*
+ * XSAVE/XRSTOR
+ *     save/restore GSSE/SIMD/MMX fpu state
+ *
+ * The pointer passed in must be 64-byte aligned.
+ * See above comment for more information.
+ */
+#if defined(__GNUC__) && (defined(VMM) || defined(VMKERNEL) || defined(FROBOS))
+
+static INLINE void 
+XSAVE_ES1(void *save, uint64 mask)
+{
+   __asm__ __volatile__ (
+        "xsave %0 \n"
+        : "=m" (*(uint8 *)save)
+        : "a" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "memory");
+}
+
+static INLINE void 
+XSAVEOPT_ES1(void *save, uint64 mask)
+{
+   __asm__ __volatile__ (
+        "xsaveopt %0 \n"
+        : "=m" (*(uint8 *)save)
+        : "a" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "memory");
+}
+
+static INLINE void 
+XRSTOR_ES1(const void *load, uint64 mask)
+{
+   __asm__ __volatile__ (
+        "xrstor %0 \n"
+        :
+        : "m" (*(const uint8 *)load),
+          "a" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "memory");
+}
+
+static INLINE void 
+XRSTOR_AMD_ES0(const void *load, uint64 mask)
+{
+   uint64 dummy = 0;
+
+   __asm__ __volatile__ 
+       ("fnstsw  %%ax    \n"     // Grab x87 ES bit
+        "bt      $7,%%ax \n"     // Test ES bit
+        "jnc     1f      \n"     // Jump if ES=0
+        "fnclex          \n"     // ES=1. Clear it so fild doesn't trap
+        "1:              \n"
+        "ffree   %%st(7) \n"     // Clear tag bit - avoid poss. stack overflow
+        "fildl   %0      \n"     // Dummy Load from "safe address" changes all
+                                 // x87 exception pointers.
+        "mov %%ebx, %%eax \n"
+        "xrstor %1 \n"
+        :
+        : "m" (dummy), "m" (*(const uint8 *)load),
+          "b" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "eax", "memory");
+}
+#endif /* __GNUC__ */
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * Div643232 --
+ *
+ *    Unsigned integer division:
+ *       The dividend is 64-bit wide
+ *       The divisor  is 32-bit wide
+ *       The quotient is 32-bit wide
+ *
+ *    Use this function if you are certain that:
+ *    o Either the quotient will fit in 32 bits,
+ *    o Or your code is ready to handle a #DE exception indicating overflow.
+ *    If that is not the case, then use Div643264().
+ *
+ * Results:
+ *    Quotient and remainder
+ *
+ * Side effects:
+ *    None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+#if defined(__GNUC__)
+
+static INLINE void
+Div643232(uint64 dividend,   // IN
+          uint32 divisor,    // IN
+          uint32 *quotient,  // OUT
+          uint32 *remainder) // OUT
+{
+   __asm__(
+      "divl %4"
+      : "=a" (*quotient),
+        "=d" (*remainder)
+      : "0" ((uint32)dividend),
+        "1" ((uint32)(dividend >> 32)),
+        "rm" (divisor)
+      : "cc"
+   );
+}
+
+#elif defined _MSC_VER
+
+static INLINE void
+Div643232(uint64 dividend,   // IN
+          uint32 divisor,    // IN
+          uint32 *quotient,  // OUT
+          uint32 *remainder) // OUT
+{
+   __asm {
+      mov  eax, DWORD PTR [dividend]
+      mov  edx, DWORD PTR [dividend+4]
+      div  DWORD PTR [divisor]
+      mov  edi, DWORD PTR [quotient]
+      mov  [edi], eax
+      mov  edi, DWORD PTR [remainder]
+      mov  [edi], edx
+   }
+}
+
+#else
+#error No compiler defined for Div643232
+#endif
+
+
+#if defined(__GNUC__)
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * Div643264 --
+ *
+ *    Unsigned integer division:
+ *       The dividend is 64-bit wide
+ *       The divisor  is 32-bit wide
+ *       The quotient is 64-bit wide
+ *
+ * Results:
+ *    Quotient and remainder
+ *
+ * Side effects:
+ *    None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE void
+Div643264(uint64 dividend,   // IN
+          uint32 divisor,    // IN
+          uint64 *quotient,  // OUT
+          uint32 *remainder) // OUT
+{
+   uint32 hQuotient;
+   uint32 lQuotient;
+
+   __asm__(
+      "divl %5"        "\n\t"
+      "movl %%eax, %0" "\n\t"
+      "movl %4, %%eax" "\n\t"
+      "divl %5"
+      : "=&rm" (hQuotient),
+        "=a" (lQuotient),
+        "=d" (*remainder)
+      : "1" ((uint32)(dividend >> 32)),
+        "g" ((uint32)dividend),
+        "rm" (divisor),
+        "2" (0)
+      : "cc"
+   );
+   *quotient = (uint64)hQuotient << 32 | lQuotient;
+}
+#endif
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * Mul64x3264 --
+ *
+ *    Unsigned integer by fixed point multiplication, with rounding:
+ *       result = floor(multiplicand * multiplier * 2**(-shift) + 0.5)
+ * 
+ *       Unsigned 64-bit integer multiplicand.
+ *       Unsigned 32-bit fixed point multiplier, represented as
+ *         (multiplier, shift), where shift < 64.
+ *
+ * Result:
+ *       Unsigned 64-bit integer product.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+#if defined(__GNUC__) && \
+   (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 4)) && \
+   !defined(MUL64_NO_ASM)
+
+static INLINE uint64
+Mul64x3264(uint64 multiplicand, uint32 multiplier, uint32 shift)
+{
+   uint64 result;
+   uint32 tmp1, tmp2;
+   // ASSERT(shift >= 0 && shift < 64);
+  
+   __asm__("mov   %%eax, %2\n\t"      // Save lo(multiplicand)
+           "mov   %%edx, %%eax\n\t"   // Get hi(multiplicand)
+           "mull  %4\n\t"             // p2 = hi(multiplicand) * multiplier
+           "xchg  %%eax, %2\n\t"      // Save lo(p2), get lo(multiplicand)
+           "mov   %%edx, %1\n\t"      // Save hi(p2)
+           "mull  %4\n\t"             // p1 = lo(multiplicand) * multiplier
+           "addl  %2, %%edx\n\t"      // hi(p1) += lo(p2)
+           "adcl  $0, %1\n\t"         // hi(p2) += carry from previous step
+           "cmpl  $32, %%ecx\n\t"     // shift < 32?
+           "jl    2f\n\t"             // Go if so
+           "shll  $1, %%eax\n\t"      // Save lo(p1) bit 31 in CF in case shift=32
+           "mov   %%edx, %%eax\n\t"   // result = hi(p2):hi(p1) >> (shift & 31)
+           "mov   %1, %%edx\n\t"
+           "shrdl %%edx, %%eax\n\t"
+           "mov   $0, %2\n\t"
+           "adcl  $0, %2\n\t"         // Get highest order bit shifted out, from CF
+           "shrl  %%cl, %%edx\n\t"
+           "jmp   3f\n"
+        "2:\n\t"
+           "xor   %2, %2\n\t"
+           "shrdl %%edx, %%eax\n\t"   // result = hi(p2):hi(p1):lo(p1) >> shift
+           "adcl  $0, %2\n\t"         // Get highest order bit shifted out, from CF
+           "shrdl %1, %%edx\n"
+        "3:\n\t"
+           "addl  %2, %%eax\n\t"      // result += highest order bit shifted out
+           "adcl  $0, %%edx"
+           : "=A" (result), "=&r" (tmp1), "=&r" (tmp2)
+           : "0" (multiplicand), "rm" (multiplier), "c" (shift)
+           : "cc");
+   return result;
+}
+
+#elif defined _MSC_VER
+#pragma warning(disable: 4035)
+
+static INLINE uint64
+Mul64x3264(uint64 multiplicand, uint32 multiplier, uint32 shift)
+{
+   // ASSERT(shift >= 0 && shift < 64);
+
+   __asm {
+      mov  eax, DWORD PTR [multiplicand+4]  // Get hi(multiplicand)
+      mul  DWORD PTR [multiplier]           // p2 = hi(multiplicand) * multiplier
+      mov  ecx, eax                         // Save lo(p2)
+      mov  ebx, edx                         // Save hi(p2)
+      mov  eax, DWORD PTR [multiplicand]    // Get lo(multiplicand)
+      mul  DWORD PTR [multiplier+0]         // p1 = lo(multiplicand) * multiplier
+      add  edx, ecx                         // hi(p1) += lo(p2)
+      adc  ebx, 0                           // hi(p2) += carry from previous step
+      mov  ecx, DWORD PTR [shift]           // Get shift
+      cmp  ecx, 32                          // shift < 32?
+      jl   SHORT l2                         // Go if so
+      shl  eax, 1                           // Save lo(p1) bit 31 in CF in case shift=32
+      mov  eax, edx                         // result = hi(p2):hi(p1) >> (shift & 31)
+      mov  edx, ebx
+      shrd eax, edx, cl
+      mov  esi, 0
+      adc  esi, 0                           // Get highest order bit shifted out, from CF
+      shr  edx, cl
+      jmp  SHORT l3
+   l2:
+      Xor  esi, esi
+      shrd eax, edx, cl                     // result = hi(p2):hi(p1):lo(p1) >> shift
+      adc  esi, 0                           // Get highest order bit shifted out, from CF
+      shrd edx, ebx, cl
+   l3:
+      add  eax, esi                         // result += highest order bit shifted out
+      adc  edx, 0
+   }
+   // return with result in edx:eax
+}
+
+#pragma warning(default: 4035)
+#else
+#define MUL64_NO_ASM 1
+#include "mul64.h"
+#endif
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * Muls64x32s64 --
+ *
+ *    Signed integer by fixed point multiplication, with rounding:
+ *       result = floor(multiplicand * multiplier * 2**(-shift) + 0.5)
+ * 
+ *       Signed 64-bit integer multiplicand.
+ *       Unsigned 32-bit fixed point multiplier, represented as
+ *         (multiplier, shift), where shift < 64.
+ *
+ * Result:
+ *       Signed 64-bit integer product.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+#if defined(__GNUC__) && \
+   (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 4)) && \
+   !defined(MUL64_NO_ASM)
+
+static INLINE int64
+Muls64x32s64(int64 multiplicand, uint32 multiplier, uint32 shift)
+{
+   int64 result;
+   uint32 tmp1, tmp2;
+   // ASSERT(shift >= 0 && shift < 64);
+
+   __asm__("mov   %%eax, %2\n\t"      // Save lo(multiplicand)
+           "mov   %%edx, %%eax\n\t"   // Get hi(multiplicand)
+           "test  %%eax, %%eax\n\t"   // Check sign of multiplicand
+           "jl    0f\n\t"             // Go if negative
+           "mull  %4\n\t"             // p2 = hi(multiplicand) * multiplier
+           "jmp   1f\n"
+        "0:\n\t"
+           "mull  %4\n\t"             // p2 = hi(multiplicand) * multiplier
+           "sub   %4, %%edx\n"        // hi(p2) += -1 * multiplier
+        "1:\n\t"
+           "xchg  %%eax, %2\n\t"      // Save lo(p2), get lo(multiplicand)
+           "mov   %%edx, %1\n\t"      // Save hi(p2)
+           "mull  %4\n\t"             // p1 = lo(multiplicand) * multiplier
+           "addl  %2, %%edx\n\t"      // hi(p1) += lo(p2)
+           "adcl  $0, %1\n\t"         // hi(p2) += carry from previous step
+           "cmpl  $32, %%ecx\n\t"     // shift < 32?
+           "jl    2f\n\t"             // Go if so
+           "shll  $1, %%eax\n\t"      // Save lo(p1) bit 31 in CF in case shift=32
+           "mov   %%edx, %%eax\n\t"   // result = hi(p2):hi(p1) >> (shift & 31)
+           "mov   %1, %%edx\n\t"
+           "shrdl %%edx, %%eax\n\t"
+           "mov   $0, %2\n\t"
+           "adcl  $0, %2\n\t"         // Get highest order bit shifted out from CF
+           "sarl  %%cl, %%edx\n\t"
+           "jmp   3f\n"
+        "2:\n\t"
+           "xor   %2, %2\n\t"
+           "shrdl %%edx, %%eax\n\t"   // result = hi(p2):hi(p1):lo(p1) >> shift
+           "adcl  $0, %2\n\t"         // Get highest order bit shifted out from CF
+           "shrdl %1, %%edx\n"
+        "3:\n\t"
+           "addl  %2, %%eax\n\t"      // result += highest order bit shifted out
+           "adcl  $0, %%edx"
+           : "=A" (result), "=&r" (tmp1), "=&rm" (tmp2)
+           : "0" (multiplicand), "rm" (multiplier), "c" (shift)
+           : "cc");
+   return result;
+}
+
+#elif defined(_MSC_VER)
+#pragma warning(disable: 4035)
+
+static INLINE int64
+Muls64x32s64(int64 multiplicand, uint32 multiplier, uint32 shift)
+{
+   //ASSERT(shift >= 0 && shift < 64);
+  
+   __asm {
+      mov  eax, DWORD PTR [multiplicand+4]  // Get hi(multiplicand)
+      test eax, eax                         // Check sign of multiplicand
+      jl   SHORT l0                         // Go if negative
+      mul  DWORD PTR [multiplier]           // p2 = hi(multiplicand) * multiplier
+      jmp  SHORT l1
+   l0:
+      mul  DWORD PTR [multiplier]           // p2 = hi(multiplicand) * multiplier
+      sub  edx, DWORD PTR [multiplier]      // hi(p2) += -1 * multiplier
+   l1:
+      mov  ecx, eax                         // Save lo(p2)
+      mov  ebx, edx                         // Save hi(p2)
+      mov  eax, DWORD PTR [multiplicand]    // Get lo(multiplicand)
+      mul  DWORD PTR [multiplier]           // p1 = lo(multiplicand) * multiplier
+      add  edx, ecx                         // hi(p1) += lo(p2)
+      adc  ebx, 0                           // hi(p2) += carry from previous step
+      mov  ecx, DWORD PTR [shift]           // Get shift
+      cmp  ecx, 32                          // shift < 32?
+      jl   SHORT l2                         // Go if so
+      shl  eax, 1                           // Save lo(p1) bit 31 in CF in case shift=32
+      mov  eax, edx                         // result = hi(p2):hi(p1) >> (shift & 31)
+      mov  edx, ebx
+      shrd eax, edx, cl
+      mov  esi, 0
+      adc  esi, 0                           // Get highest order bit shifted out, from CF
+      sar  edx, cl
+      jmp  SHORT l3
+   l2:
+      Xor  esi, esi
+      shrd eax, edx, cl                     // result = hi(p2):hi(p1):lo(p1) << shift
+      adc  esi, 0                           // Get highest order bit shifted out, from CF
+      shrd edx, ebx, cl
+   l3:
+      add  eax, esi                         // result += highest order bit shifted out
+      adc  edx, 0
+   }
+   // return with result in edx:eax
+}
+
+#pragma warning(default: 4035)
+#endif
+
+
+#if defined __cplusplus
+} // extern "C"
+#endif
+
+#endif // _VM_BASIC_ASM_X86_H_

vmmon-only/include/vm_basic_asm_x86_64.h

@@ -0,0 +1,651 @@
+/*********************************************************
+ * Copyright (C) 1998-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vm_basic_asm_x86_64.h
+ *
+ *	Basic x86_64 asm macros.
+ */
+
+#ifndef _VM_BASIC_ASM_X86_64_H_
+#define _VM_BASIC_ASM_X86_64_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#ifndef VM_X86_64
+#error "This file is x86-64 only!"
+#endif
+
+#if defined(_MSC_VER) && !defined(BORA_NO_WIN32_INTRINS)
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+uint64 _umul128(uint64 multiplier, uint64 multiplicand,
+                uint64 *highProduct);
+int64 _mul128(int64 multiplier, int64 multiplicand,
+              int64 *highProduct);
+uint64 __shiftright128(uint64 lowPart, uint64 highPart, uint8 shift);
+#ifdef ULM
+void _fxsave64(void *save);
+void _fxsave(void *save);
+void _fxrstor64(const void *load);
+void _fxrstor(const void *load);
+void _xsave64(void *save, uint64 mask);
+void _xsave(void *save, uint64 mask);
+void _xsaveopt64(void *save, uint64 mask);
+void _xsavec(void *save, uint64 mask);
+void _xrstor64(const void *load, uint64 mask);
+void _xrstor(const void *load, uint64 mask);
+#endif /* ULM */
+#ifdef __cplusplus
+}
+#endif
+
+#pragma intrinsic(_umul128, _mul128, __shiftright128)
+
+#ifdef ULM
+#pragma intrinsic(_fxsave64, _fxsave, _fxrstor64, _fxrstor, _xsave64, _xsave, \
+                  _xsaveopt64, _xsavec, _xrstor64, _xrstor)
+#endif /* ULM */
+
+#endif // _MSC_VER
+
+#if defined(__GNUC__)
+/*
+ * GET_CURRENT_PC
+ *
+ * Returns the current program counter (i.e. instruction pointer i.e. rip
+ * register on x86_64). In the example below:
+ *
+ *   foo.c
+ *   L123: Foo(GET_CURRENT_PC())
+ *
+ * the return value from GET_CURRENT_PC will point a debugger to L123.
+ */
+#define GET_CURRENT_PC() ({                                           \
+      void *__rip;                                                    \
+      asm("lea 0(%%rip), %0;\n\t"                                     \
+         : "=r" (__rip));                                             \
+      __rip;                                                          \
+})
+
+/*
+ * GET_CURRENT_LOCATION
+ *
+ * Updates the arguments with the values of the %rip, %rbp, and %rsp
+ * registers at the current code location where the macro is invoked,
+ * and the return address.
+ */
+#define GET_CURRENT_LOCATION(rip, rbp, rsp, retAddr)  do {         \
+      asm("lea 0(%%rip), %0\n"                                     \
+          "mov %%rbp, %1\n"                                        \
+          "mov %%rsp, %2\n"                                        \
+          : "=r" (rip), "=r" (rbp), "=r" (rsp));                   \
+      retAddr = (uint64) GetReturnAddress();                       \
+   } while (0)
+#endif
+
+/*
+ * FXSAVE/FXRSTOR
+ *     save/restore SIMD/MMX fpu state
+ *
+ * The pointer passed in must be 16-byte aligned.
+ *
+ * Intel and AMD processors behave differently w.r.t. fxsave/fxrstor. Intel
+ * processors unconditionally save the exception pointer state (instruction
+ * ptr., data ptr., and error instruction opcode). FXSAVE_ES1 and FXRSTOR_ES1
+ * work correctly for Intel processors.
+ *
+ * AMD processors only save the exception pointer state if ES=1. This leads to a
+ * security hole whereby one process/VM can inspect the state of another process
+ * VM. The AMD recommended workaround involves clobbering the exception pointer
+ * state unconditionally, and this is implemented in FXRSTOR_AMD_ES0. Note that
+ * FXSAVE_ES1 will only save the exception pointer state for AMD processors if
+ * ES=1.
+ *
+ * The workaround (FXRSTOR_AMD_ES0) only costs 1 cycle more than just doing an
+ * fxrstor, on both AMD Opteron and Intel Core CPUs.
+ */
+
+#if defined(VMM) || defined(VMKERNEL) || defined(FROBOS) || defined(ULM)
+static INLINE void
+FXSAVE_ES1(void *save)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__ ("fxsaveq %0  \n" : "=m" (*(uint8 *)save) : : "memory");
+#elif defined(_MSC_VER)
+   _fxsave64(save);
+#endif
+}
+
+static INLINE void
+FXSAVE_COMPAT_ES1(void *save)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__ ("fxsave %0  \n" : "=m" (*(uint8 *)save) : : "memory");
+#elif defined(_MSC_VER)
+   _fxsave(save);
+#endif
+}
+
+static INLINE void
+FXRSTOR_ES1(const void *load)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__ ("fxrstorq %0 \n"
+                         : : "m" (*(const uint8 *)load) : "memory");
+#elif defined(_MSC_VER)
+   _fxrstor64(load);
+#endif
+}
+
+static INLINE void
+FXRSTOR_COMPAT_ES1(const void *load)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__ ("fxrstor %0 \n"
+                         : : "m" (*(const uint8 *)load) : "memory");
+#elif defined(_MSC_VER)
+   _fxrstor(load);
+#endif
+}
+
+#if defined(__GNUC__)
+static INLINE void
+FXRSTOR_AMD_ES0(const void *load)
+{
+   uint64 dummy = 0;
+
+   __asm__ __volatile__
+       ("fnstsw  %%ax    \n"     // Grab x87 ES bit
+        "bt      $7,%%ax \n"     // Test ES bit
+        "jnc     1f      \n"     // Jump if ES=0
+        "fnclex          \n"     // ES=1. Clear it so fild doesn't trap
+        "1:              \n"
+        "ffree   %%st(7) \n"     // Clear tag bit - avoid poss. stack overflow
+        "fildl   %0      \n"     // Dummy Load from "safe address" changes all
+                                 // x87 exception pointers.
+        "fxrstorq %1 \n"
+        :
+        : "m" (dummy), "m" (*(const uint8 *)load)
+        : "ax", "memory");
+}
+
+#endif /* __GNUC__ */
+
+/*
+ * XSAVE/XRSTOR
+ *     save/restore GSSE/SIMD/MMX fpu state
+ *
+ * The pointer passed in must be 64-byte aligned.
+ * See above comment for more information.
+ */
+
+static INLINE void
+XSAVE_ES1(void *save, uint64 mask)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__ (
+        "xsaveq %0 \n"
+        : "=m" (*(uint8 *)save)
+        : "a" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "memory");
+#elif defined(_MSC_VER)
+   _xsave64(save, mask);
+#endif
+}
+
+static INLINE void
+XSAVE_COMPAT_ES1(void *save, uint64 mask)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__ (
+        "xsave %0 \n"
+        : "=m" (*(uint8 *)save)
+        : "a" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "memory");
+#elif defined(_MSC_VER)
+   _xsave(save, mask);
+#endif
+}
+
+static INLINE void
+XSAVEOPT_ES1(void *save, uint64 mask)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__ (
+        "xsaveoptq %0 \n"
+        : "=m" (*(uint8 *)save)
+        : "a" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "memory");
+#elif defined(_MSC_VER)
+   _xsaveopt64(save, mask);
+#endif
+}
+
+static INLINE void
+XSAVEC_COMPAT_ES1(void *save, uint64 mask)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__ (
+        "xsavec %0 \n"
+        : "=m" (*(uint8 *)save)
+        : "a" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "memory");
+#elif defined(_MSC_VER)
+   _xsavec(save, mask);
+#endif
+}
+
+static INLINE void
+XRSTOR_ES1(const void *load, uint64 mask)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__ (
+        "xrstorq %0 \n"
+        :
+        : "m" (*(const uint8 *)load),
+          "a" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "memory");
+#elif defined(_MSC_VER)
+   _xrstor64(load, mask);
+#endif
+}
+
+static INLINE void
+XRSTOR_COMPAT_ES1(const void *load, uint64 mask)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__ (
+        "xrstor %0 \n"
+        :
+        : "m" (*(const uint8 *)load),
+          "a" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "memory");
+#elif defined(_MSC_VER)
+   _xrstor(load, mask);
+#endif
+}
+
+#if defined(__GNUC__)
+static INLINE void
+XRSTOR_AMD_ES0(const void *load, uint64 mask)
+{
+   uint64 dummy = 0;
+
+   __asm__ __volatile__
+       ("fnstsw  %%ax    \n"     // Grab x87 ES bit
+        "bt      $7,%%ax \n"     // Test ES bit
+        "jnc     1f      \n"     // Jump if ES=0
+        "fnclex          \n"     // ES=1. Clear it so fild doesn't trap
+        "1:              \n"
+        "ffree   %%st(7) \n"     // Clear tag bit - avoid poss. stack overflow
+        "fildl   %0      \n"     // Dummy Load from "safe address" changes all
+                                 // x87 exception pointers.
+        "mov %%ebx, %%eax \n"
+        "xrstorq %1 \n"
+        :
+        : "m" (dummy), "m" (*(const uint8 *)load),
+          "b" ((uint32)mask), "d" ((uint32)(mask >> 32))
+        : "eax", "memory");
+}
+
+#endif /* __GNUC__ */
+#endif /* VMM || VMKERNEL || FROBOS || ULM */
+
+/*
+ * XTEST
+ *     Return TRUE if processor is in transaction region.
+ *
+ *  Using condition codes as output values (=@ccnz) requires gcc6 or
+ *  above.  Clang does not support condition codes as output
+ *  constraints.
+ *
+ */
+#if defined(__GNUC__) && (defined(VMM) || defined(VMKERNEL) || defined(FROBOS))
+static INLINE Bool
+xtest(void)
+{
+   Bool result;
+#if defined(__clang__)
+   __asm__ __volatile__("xtest\n"
+                        "setnz %%al"
+                        : "=a" (result) : : "cc");
+#else
+   __asm__ __volatile__("xtest"
+                        : "=@ccnz" (result) : : "cc");
+#endif
+   return result;
+}
+
+#endif /* __GNUC__ */
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * Mul64x6464 --
+ *
+ *    Unsigned integer by fixed point multiplication, with rounding:
+ *       result = floor(multiplicand * multiplier * 2**(-shift) + 0.5)
+ *
+ *       Unsigned 64-bit integer multiplicand.
+ *       Unsigned 64-bit fixed point multiplier, represented as
+ *         (multiplier, shift), where shift < 64.
+ *
+ * Result:
+ *       Unsigned 64-bit integer product.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+#if defined(__GNUC__) && !defined(MUL64_NO_ASM)
+
+static INLINE uint64
+Mul64x6464(uint64 multiplicand,
+           uint64 multiplier,
+           uint32 shift)
+{
+   /*
+    * Implementation:
+    *    Multiply 64x64 bits to yield a full 128-bit product.
+    *    Clear the carry bit (needed for the shift == 0 case).
+    *    Shift result in RDX:RAX right by "shift".
+    *    Add the carry bit.  (If shift > 0, this is the highest order bit
+    *      that was discarded by the shift; else it is 0.)
+    *    Return the low-order 64 bits of the above.
+    *
+    */
+   uint64 result, dummy;
+
+   __asm__("mulq    %3           \n\t"
+           "clc                  \n\t"
+           "shrdq   %b4, %1, %0  \n\t"
+           "adc     $0, %0       \n\t"
+           : "=a" (result),
+             "=d" (dummy)
+           : "0"  (multiplier),
+             "rm" (multiplicand),
+             "c"  (shift)
+           : "cc");
+   return result;
+}
+
+#elif defined(_MSC_VER) && !defined(MUL64_NO_ASM)
+
+static INLINE uint64
+Mul64x6464(uint64 multiplicand,
+           uint64 multiplier,
+           uint32 shift)
+{
+   /*
+    * Unfortunately, MSVC intrinsics don't give us access to the carry
+    * flag after a 128-bit shift, so the implementation is more
+    * awkward:
+    *    Multiply 64x64 bits to yield a full 128-bit product.
+    *    Shift result right by "shift".
+    *    If shift != 0, extract and add in highest order bit that was
+    *      discarded by the shift.
+    *    Return the low-order 64 bits of the above.
+    */
+   uint64 tmplo, tmphi;
+   tmplo = _umul128(multiplicand, multiplier, &tmphi);
+   if (shift == 0) {
+      return tmplo;
+   } else {
+      return __shiftright128(tmplo, tmphi, (uint8) shift) +
+         ((tmplo >> (shift - 1)) & 1);
+   }
+}
+
+#else
+#define MUL64_NO_ASM 1
+#include "mul64.h"
+#endif
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * Muls64x64s64 --
+ *
+ *    Signed integer by fixed point multiplication, with rounding:
+ *       result = floor(multiplicand * multiplier * 2**(-shift) + 0.5)
+ *
+ *       Signed 64-bit integer multiplicand.
+ *       Unsigned 64-bit fixed point multiplier, represented as
+ *         (multiplier, shift), where shift < 64.
+ *
+ * Result:
+ *       Signed 64-bit integer product.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+#if defined(__GNUC__) && !defined(MUL64_NO_ASM)
+
+static inline int64
+Muls64x64s64(int64 multiplicand,
+             int64 multiplier,
+             uint32 shift)
+{
+   int64 result, dummy;
+
+   /* Implementation:
+    *    Multiply 64x64 bits to yield a full 128-bit product.
+    *    Clear the carry bit (needed for the shift == 0 case).
+    *    Shift result in RDX:RAX right by "shift".
+    *    Add the carry bit.  (If shift > 0, this is the highest order bit
+    *      that was discarded by the shift; else it is 0.)
+    *    Return the low-order 64 bits of the above.
+    *
+    *    Note: using the unsigned shrd instruction is correct because
+    *    shift < 64 and we return only the low 64 bits of the shifted
+    *    result.
+    */
+   __asm__("imulq   %3           \n\t"
+           "clc                  \n\t"
+           "shrdq   %b4, %1, %0  \n\t"
+           "adc     $0, %0       \n\t"
+           : "=a" (result),
+             "=d" (dummy)
+           : "0"  (multiplier),
+             "rm" (multiplicand),
+             "c"  (shift)
+           : "cc");
+   return result;
+}
+
+#elif defined(_MSC_VER) && !defined(MUL64_NO_ASM)
+
+static INLINE int64
+Muls64x64s64(int64 multiplicand,
+             int64 multiplier,
+             uint32 shift)
+{
+   /*
+    * Unfortunately, MSVC intrinsics don't give us access to the carry
+    * flag after a 128-bit shift, so the implementation is more
+    * awkward:
+    *    Multiply 64x64 bits to yield a full 128-bit product.
+    *    Shift result right by "shift".
+    *    If shift != 0, extract and add in highest order bit that was
+    *      discarded by the shift.
+    *    Return the low-order 64 bits of the above.
+    *
+    * Note: using an unsigned shift is correct because shift < 64 and
+    * we return only the low 64 bits of the shifted result.
+    */
+   int64 tmplo, tmphi;
+   tmplo = _mul128(multiplicand, multiplier, &tmphi);
+   if (shift == 0) {
+      return tmplo;
+   } else {
+      return __shiftright128(tmplo, tmphi, (uint8) shift) +
+         ((tmplo >> (shift - 1)) & 1);
+   }
+}
+
+#endif
+
+#ifndef MUL64_NO_ASM
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * Mul64x3264 --
+ *
+ *    Unsigned integer by fixed point multiplication, with rounding:
+ *       result = floor(multiplicand * multiplier * 2**(-shift) + 0.5)
+ *
+ *       Unsigned 64-bit integer multiplicand.
+ *       Unsigned 32-bit fixed point multiplier, represented as
+ *         (multiplier, shift), where shift < 64.
+ *
+ * Result:
+ *       Unsigned 64-bit integer product.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE uint64
+Mul64x3264(uint64 multiplicand, uint32 multiplier, uint32 shift)
+{
+   return Mul64x6464(multiplicand, multiplier, shift);
+}
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * Muls64x32s64 --
+ *
+ *    Signed integer by fixed point multiplication, with rounding:
+ *       result = floor(multiplicand * multiplier * 2**(-shift) + 0.5)
+ *
+ *       Signed 64-bit integer multiplicand.
+ *       Unsigned 32-bit fixed point multiplier, represented as
+ *         (multiplier, shift), where shift < 64.
+ *
+ * Result:
+ *       Signed 64-bit integer product.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE int64
+Muls64x32s64(int64 multiplicand, uint32 multiplier, uint32 shift)
+{
+   return Muls64x64s64(multiplicand, multiplier, shift);
+}
+#endif
+
+#if defined(__GNUC__)
+
+static INLINE void *
+uint64set(void *dst, uint64 val, uint64 count)
+{
+   int dummy0;
+   int dummy1;
+   __asm__ __volatile__("\t"
+                        "cld"            "\n\t"
+                        "rep ; stosq"    "\n"
+                        : "=c" (dummy0), "=D" (dummy1)
+                        : "0" (count), "1" (dst), "a" (val)
+                        : "memory", "cc");
+   return dst;
+}
+
+#endif
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * Div643232 --
+ *
+ *    Unsigned integer division:
+ *       The dividend is 64-bit wide
+ *       The divisor  is 32-bit wide
+ *       The quotient is 32-bit wide
+ *
+ *    Use this function if you are certain that the quotient will fit in 32 bits,
+ *    If that is not the case, a #DE exception was generated in 32-bit version,
+ *    but not in this 64-bit version. So please be careful.
+ *
+ * Results:
+ *    Quotient and remainder
+ *
+ * Side effects:
+ *    None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+#if defined(__GNUC__) || defined(_MSC_VER)
+
+static INLINE void
+Div643232(uint64 dividend,   // IN
+          uint32 divisor,    // IN
+          uint32 *quotient,  // OUT
+          uint32 *remainder) // OUT
+{
+   *quotient = (uint32)(dividend / divisor);
+   *remainder = (uint32)(dividend % divisor);
+}
+
+#endif
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * Div643264 --
+ *
+ *    Unsigned integer division:
+ *       The dividend is 64-bit wide
+ *       The divisor  is 32-bit wide
+ *       The quotient is 64-bit wide
+ *
+ * Results:
+ *    Quotient and remainder
+ *
+ * Side effects:
+ *    None
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+#if defined(__GNUC__)
+
+static INLINE void
+Div643264(uint64 dividend,   // IN
+          uint32 divisor,    // IN
+          uint64 *quotient,  // OUT
+          uint32 *remainder) // OUT
+{
+   *quotient = dividend / divisor;
+   *remainder = dividend % divisor;
+}
+
+#endif
+
+#endif // _VM_BASIC_ASM_X86_64_H_

vmmon-only/include/vm_basic_asm_x86_common.h

@@ -0,0 +1,790 @@
+/*********************************************************
+ * Copyright (C) 2013-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vm_basic_asm_x86_common.h --
+ *
+ *	Basic assembler macros common to 32-bit and 64-bit x86 ISA.
+ */
+
+#ifndef _VM_BASIC_ASM_X86_COMMON_H_
+#define _VM_BASIC_ASM_X86_COMMON_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#ifndef VM_X86_ANY
+#error "Should be included only in x86 builds"
+#endif
+
+/*
+ * x86-64 windows doesn't support inline asm so we have to use these
+ * intrinsic functions defined in the compiler.  Not all of these are well
+ * documented.  There is an array in the compiler dll (c1.dll) which has
+ * an array of the names of all the intrinsics minus the leading
+ * underscore.  Searching around in the ntddk.h file can also be helpful.
+ *
+ * The declarations for the intrinsic functions were taken from the DDK.
+ * Our declarations must match the ddk's otherwise the 64-bit c++ compiler
+ * will complain about second linkage of the intrinsic functions.
+ * We define the intrinsic using the basic types corresponding to the
+ * Windows typedefs. This avoids having to include windows header files
+ * to get to the windows types.
+ */
+#if defined(_MSC_VER) && !defined(BORA_NO_WIN32_INTRINS)
+#ifdef __cplusplus
+extern "C" {
+#endif
+/*
+ * It seems x86 & x86-64 windows still implements these intrinsic
+ * functions.  The documentation for the x86-64 suggest the
+ * __inbyte/__outbyte intrinsics even though the _in/_out work fine and
+ * __inbyte/__outbyte aren't supported on x86.
+ */
+int            _inp(unsigned short);
+unsigned short _inpw(unsigned short);
+unsigned long  _inpd(unsigned short);
+
+int            _outp(unsigned short, int);
+unsigned short _outpw(unsigned short, unsigned short);
+unsigned long  _outpd(uint16, unsigned long);
+#pragma intrinsic(_inp, _inpw, _inpd, _outp, _outpw, _outpw, _outpd)
+
+/*
+ * Prevents compiler from re-ordering reads, writes and reads&writes.
+ * These functions do not add any instructions thus only affect
+ * the compiler ordering.
+ *
+ * See:
+ * `Lockless Programming Considerations for Xbox 360 and Microsoft Windows'
+ * http://msdn.microsoft.com/en-us/library/bb310595(VS.85).aspx
+ */
+void _ReadBarrier(void);
+void _WriteBarrier(void);
+void _ReadWriteBarrier(void);
+#pragma intrinsic(_ReadBarrier, _WriteBarrier, _ReadWriteBarrier)
+
+void _mm_mfence(void);
+void _mm_lfence(void);
+#pragma intrinsic(_mm_mfence, _mm_lfence)
+
+long _InterlockedXor(long volatile *, long);
+#pragma intrinsic(_InterlockedXor)
+
+unsigned int __getcallerseflags(void);
+#pragma intrinsic(__getcallerseflags)
+
+#ifdef VM_X86_64
+/*
+ * intrinsic functions only supported by x86-64 windows as of 2k3sp1
+ */
+unsigned __int64 __rdtsc(void);
+void             __stosw(unsigned short *, unsigned short, size_t);
+void             __stosd(unsigned long *, unsigned long, size_t);
+void             _mm_pause(void);
+#pragma intrinsic(__rdtsc, __stosw, __stosd, _mm_pause)
+
+unsigned char  _BitScanForward64(unsigned long *, unsigned __int64);
+unsigned char  _BitScanReverse64(unsigned long *, unsigned __int64);
+#pragma intrinsic(_BitScanForward64, _BitScanReverse64)
+#endif /* VM_X86_64 */
+
+unsigned char  _BitScanForward(unsigned long *, unsigned long);
+unsigned char  _BitScanReverse(unsigned long *, unsigned long);
+#pragma intrinsic(_BitScanForward, _BitScanReverse)
+
+unsigned char  _bittest(const long *, long);
+unsigned char  _bittestandset(long *, long);
+unsigned char  _bittestandreset(long *, long);
+unsigned char  _bittestandcomplement(long *, long);
+#pragma intrinsic(_bittest, _bittestandset, _bittestandreset, _bittestandcomplement)
+#ifdef VM_X86_64
+unsigned char  _bittestandset64(__int64 *, __int64);
+unsigned char  _bittestandreset64(__int64 *, __int64);
+#pragma intrinsic(_bittestandset64, _bittestandreset64)
+#endif // VM_X86_64
+#ifdef __cplusplus
+}
+#endif
+#endif // _MSC_VER
+
+#ifdef __GNUC__
+/*
+ * Checked against the Intel manual and GCC --hpreg
+ *
+ * volatile because reading from port can modify the state of the underlying
+ * hardware.
+ *
+ * Note: The undocumented %z construct doesn't work (internal compiler error)
+ *       with gcc-2.95.1
+ */
+
+#define __GCC_IN(s, type, name) \
+static INLINE type              \
+name(uint16 port)               \
+{                               \
+   type val;                    \
+                                \
+   __asm__ __volatile__(        \
+      "in" #s " %w1, %0"        \
+      : "=a" (val)              \
+      : "Nd" (port)             \
+   );                           \
+                                \
+   return val;                  \
+}
+
+__GCC_IN(b, uint8, INB)
+__GCC_IN(w, uint16, INW)
+__GCC_IN(l, uint32, IN32)
+
+
+/*
+ * Checked against the Intel manual and GCC --hpreg
+ *
+ * Note: The undocumented %z construct doesn't work (internal compiler error)
+ *       with gcc-2.95.1
+ */
+
+#define __GCC_OUT(s, s2, port, val) do { \
+   __asm__(                              \
+      "out" #s " %" #s2 "1, %w0"         \
+      :                                  \
+      : "Nd" (port), "a" (val)           \
+   );                                    \
+} while (0)
+
+#define OUTB(port, val) __GCC_OUT(b, b, port, val)
+#define OUTW(port, val) __GCC_OUT(w, w, port, val)
+#define OUT32(port, val) __GCC_OUT(l, , port, val)
+
+#define GET_CURRENT_EIP(_eip) \
+      __asm__ __volatile("call 0\n\tpopl %0" : "=r" (_eip): );
+
+static INLINE unsigned int
+GetCallerEFlags(void)
+{
+   unsigned long flags;
+   asm volatile("pushf; pop %0" : "=r"(flags));
+   return flags;
+}
+
+#elif defined(_MSC_VER)
+static INLINE  uint8
+INB(uint16 port)
+{
+   return (uint8)_inp(port);
+}
+static INLINE void
+OUTB(uint16 port, uint8 value)
+{
+   _outp(port, value);
+}
+static INLINE uint16
+INW(uint16 port)
+{
+   return _inpw(port);
+}
+static INLINE void
+OUTW(uint16 port, uint16 value)
+{
+   _outpw(port, value);
+}
+static INLINE  uint32
+IN32(uint16 port)
+{
+   return _inpd(port);
+}
+static INLINE void
+OUT32(uint16 port, uint32 value)
+{
+   _outpd(port, value);
+}
+
+#ifndef VM_X86_64
+#ifdef NEAR
+#undef NEAR
+#endif
+
+#define GET_CURRENT_EIP(_eip) do { \
+   __asm call NEAR PTR $+5 \
+   __asm pop eax \
+   __asm mov _eip, eax \
+} while (0)
+#endif // VM_X86_64
+
+static INLINE unsigned int
+GetCallerEFlags(void)
+{
+   return __getcallerseflags();
+}
+
+#endif // __GNUC__
+
+/* Sequence recommended by Intel for the Pentium 4. */
+#define INTEL_MICROCODE_VERSION() (             \
+   X86MSR_SetMSR(MSR_BIOS_SIGN_ID, 0),          \
+   __GET_EAX_FROM_CPUID(1),                     \
+   X86MSR_GetMSR(MSR_BIOS_SIGN_ID))
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * MFENCE --
+ *
+ *      Wrapper around the MFENCE instruction.
+ *
+ *      Caveat Emptor! This function is _NOT_ _PORTABLE_ and most certainly
+ *      not something you should use. Take a look at the SMP_*_BARRIER_*,
+ *      DMA_*_BARRIER_* and MMIO_*_BARRIER_* interfaces instead, when writing
+ *      general OS/VMM code.
+ *
+ * Results:
+ *      None.
+ *
+ * Side effects:
+ *      See MFENCE instruction in Intel SDM or AMD Programmer's Manual.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE void
+MFENCE(void)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__(
+      "mfence"
+      ::: "memory"
+   );
+#elif defined _MSC_VER
+   _ReadWriteBarrier();
+   _mm_mfence();
+   _ReadWriteBarrier();
+#else
+#error No compiler defined for MFENCE
+#endif
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * LFENCE --
+ *
+ *      Wrapper around the LFENCE instruction.
+ *
+ *      Caveat Emptor! This function is _NOT_ _PORTABLE_ and most certainly
+ *      not something you should use. Take a look at the SMP_*_BARRIER_*,
+ *      DMA_*_BARRIER_* and MMIO_*_BARRIER_* interfaces instead, when writing
+ *      general OS/VMM code.
+ *
+ * Results:
+ *      None.
+ *
+ * Side effects:
+ *      See LFENCE instruction in Intel SDM or AMD Programmer's Manual.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE void
+LFENCE(void)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__(
+      "lfence"
+      : : : "memory"
+   );
+#elif defined _MSC_VER
+   _ReadWriteBarrier();
+   _mm_lfence();
+   _ReadWriteBarrier();
+#else
+#error No compiler defined for LFENCE
+#endif
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * SFENCE --
+ *
+ *      Wrapper around the SFENCE instruction.
+ *
+ *      Caveat Emptor! This function is _NOT_ _PORTABLE_ and most certainly
+ *      not something you should use. Take a look at the SMP_*_BARRIER_*,
+ *      DMA_*_BARRIER_* and MMIO_*_BARRIER_* interfaces instead, when writing
+ *      general OS/VMM code.
+ *
+ * Results:
+ *      None.
+ *
+ * Side effects:
+ *      See SFENCE instruction in Intel SDM or AMD Programmer's Manual.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE void
+SFENCE(void)
+{
+#ifdef __GNUC__
+   __asm__ __volatile__(
+      "sfence"
+      : : : "memory"
+   );
+#elif defined _MSC_VER
+   _ReadWriteBarrier();
+#if defined VM_X86_32
+   __asm sfence;
+#else
+   _mm_sfence();
+#endif
+   _ReadWriteBarrier();
+#else
+#error No compiler defined for SFENCE
+#endif
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * RDTSC_BARRIER --
+ *
+ *      Implements an RDTSC fence.  Instructions executed prior to the
+ *      fence will have completed before the fence and all stores to
+ *      memory are flushed from the store buffer.
+ *
+ *      On AMD, MFENCE is sufficient.  On Intel, only LFENCE is
+ *      documented to fence RDTSC, but LFENCE won't drain the store
+ *      buffer.  So, use MFENCE;LFENCE, which will work on both AMD and
+ *      Intel.
+ *
+ *      It is the callers' responsibility to check for SSE2 before
+ *      calling this function.
+ *
+ * Results:
+ *      None.
+ *
+ * Side effects:
+ *      Cause loads and stores prior to this to be globally visible, and
+ *      RDTSC will not pass.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE void
+RDTSC_BARRIER(void)
+{
+   MFENCE();
+   LFENCE();
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * LOCKED_INSN_BARRIER --
+ *
+ *      Implements a full WB load/store barrier using a locked instruction.
+ *
+ *      See PR 1674199 for details. You may choose to use this for
+ *      performance reasons over MFENCE iff you are only dealing with
+ *      WB memory accesses.
+ *
+ *      DANGER! Do not use this barrier instead of MFENCE when dealing
+ *      with non-temporal instructions or UC/WC memory accesses.
+ *
+ *      Caveat Emptor! This function is _NOT_ _PORTABLE_ and most certainly
+ *      not something you should use. Take a look at the SMP_*_BARRIER_*,
+ *      DMA_*_BARRIER_* and MMIO_*_BARRIER_* interfaces instead, when writing
+ *      general OS/VMM code.
+ *
+ * Results:
+ *      None.
+ *
+ * Side effects:
+ *      Cause WB loads and stores before the call to be globally visible
+ *      before WB loads and stores after this call.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE void
+LOCKED_INSN_BARRIER(void)
+{
+   volatile long temp = 0;
+
+#if defined __GNUC__
+   __asm__ __volatile__ (
+      "lock xorl $1, %0"
+      : "+m" (temp)
+      : /* no additional inputs */
+      : "cc", "memory");
+#elif defined _MSC_VER
+   /*
+    * Ignore warning about _InterlockedXor operation on a local variable; we are
+    * using the operation for its side-effects only.
+    */
+   #pragma warning(suppress:28113)
+   _InterlockedXor(&temp, 1);
+#else
+#error LOCKED_INSN_BARRIER not defined for this compiler
+#endif
+}
+
+/*
+ * Memory Barriers
+ * ===============
+ *
+ *    Terminology
+ *    -----------
+ *
+ * A compiler memory barrier prevents the compiler from re-ordering memory
+ * accesses accross the barrier. It is not a CPU instruction, it is a compiler
+ * directive (i.e. it does not emit any code).
+ *
+ * A CPU memory barrier prevents the CPU from re-ordering memory accesses
+ * accross the barrier. It is a CPU instruction.
+ *
+ * A memory barrier is the union of a compiler memory barrier and a CPU memory
+ * barrier. A compiler memory barrier is a useless construct by itself. It is
+ * only useful when combined with a CPU memory barrier, to implement a memory
+ * barrier.
+ *
+ *    Semantics
+ *    ---------
+ *
+ * At the time COMPILER_*_BARRIER were created (and references to them were
+ * added to the code), the code was only targetting x86. The intent of the code
+ * was really to use a memory barrier, but because x86 uses a strongly ordered
+ * memory model, the CPU would not re-order memory accesses, and the code could
+ * get away with using just a compiler memory barrier. So COMPILER_*_BARRIER
+ * were born and were implemented as compiler memory barriers _on x86_. But
+ * make no mistake, _the semantics that the code expects from
+ * COMPILER_*_BARRIER are that of a memory barrier_!
+ *
+ *    DO NOT USE!
+ *    -----------
+ *
+ * On at least one non-x86 architecture, COMPILER_*_BARRIER are
+ * 1) Misnomers
+ * 2) Not fine-grained enough to provide the best performance.
+ * For the above two reasons, usage of COMPILER_*_BARRIER is now deprecated.
+ * _Do not add new references to COMPILER_*_BARRIER._ Instead, precisely
+ * document the intent of your code by using
+ * <mem_type/purpose>_<before_access_type>_BARRIER_<after_access_type>.
+ * Existing references to COMPILER_*_BARRIER are being slowly but surely
+ * converted, and when no references are left, COMPILER_*_BARRIER will be
+ * retired.
+ *
+ * Thanks for pasting this whole comment into every architecture header.
+ */
+
+#if defined __GNUC__
+#   define COMPILER_READ_BARRIER()  COMPILER_MEM_BARRIER()
+#   define COMPILER_WRITE_BARRIER() COMPILER_MEM_BARRIER()
+#   define COMPILER_MEM_BARRIER()   __asm__ __volatile__("" ::: "memory")
+#elif defined _MSC_VER
+#   define COMPILER_READ_BARRIER()  _ReadBarrier()
+#   define COMPILER_WRITE_BARRIER() _WriteBarrier()
+#   define COMPILER_MEM_BARRIER()   _ReadWriteBarrier()
+#endif
+
+/*
+ * Memory barriers. These take the form of
+ *
+ * <mem_type/purpose>_<before_access_type>_BARRIER_<after_access_type>
+ *
+ * where:
+ *   <mem_type/purpose> is either SMP, DMA, or MMIO.
+ *   <*_access type> is either R(load), W(store) or RW(any).
+ *
+ * Above every use of these memory barriers in the code, there _must_ be a
+ * comment to justify the use, i.e. a comment which:
+ *
+ * 1) Precisely identifies which memory accesses must not be re-ordered across
+ *    the memory barrier.
+ * 2) Explains why it is important that the memory accesses not be re-ordered.
+ *
+ * Thanks for pasting this whole comment into every architecture header.
+ *
+ * This is a simplified version of Table 7-3 (Memory Access Ordering Rules) from
+ * AMD AMD64 Architecture Programmer's Manual Volume 2: System Programming
+ * (September 2018, Publication 24593, Revision 3.30).
+ *
+ * https://www.amd.com/system/files/TechDocs/24593.pdf#page=228
+ *
+ * This table only includes the memory types we care about in the context of
+ * SMP, DMA and MMIO barriers.
+ *
+ * +-------------+------+------+------+------+------+------+
+ * |\ 2nd mem op |      |      |      |      |      |      |
+ * | `---------. | R WB | R UC | R WC | W WB | W UC | W WC |
+ * | 1st mem op \|      |      |      |      |      |      |
+ * +-------------+------+------+------+------+------+------+
+ * |    R WB     |      |      | LF1  |      |      |      |
+ * +-------------+------+------+------+------+------+------+
+ * |    R UC     |      |      | LF1  |      |      |      |
+ * +-------------+------+------+------+------+------+------+
+ * |    R WC     |      |      | LF1  |      |      |      |
+ * +-------------+------+------+------+------+------+------+
+ * |    W WB     | MF1  |      | MF1  |      |      | SF2  |
+ * +-------------+------+------+------+------+------+------+
+ * |    W UC     | MF1  |      | MF1  |      |      | SF2  |
+ * +-------------+------+------+------+------+------+------+
+ * |    W WC     | MF1  |      | MF1  | SF1  |      | SF2  |
+ * +-------------+------+------+------+------+------+------+
+ *
+ * MF1 - WB or WC load may pass a previous non-conflicting WB, WC or UC store.
+ *       Use MFENCE. This is a combination of rules 'e' and 'i' in the AMD
+ *       diagram.
+ * LF1 - WC load may pass a previous WB, WC or UC load. Use LFENCE. This is
+ *       rule 'b' in the AMD diagram.
+ * SF1 - WB store may pass a previous WC store. Use SFENCE. This is rule 'j' in
+ *       the AMD diagram.
+ * SF2 - WC store may pass a previous UC, WB or non-conflicting WC store. Use
+ *       SFENCE. This is rule 'h' in the AMD diagram.
+ *
+ * To figure out the specific barrier required, pick and collapse the relevant
+ * rows and columns, choosing the strongest barrier.
+ *
+ * SMP barriers only concern with access to "normal memory" (write-back cached
+ * i.e. WB using above terminology), so we only need to worry about store-load
+ * reordering. In other cases a compiler barrier is sufficient. SMP store-load
+ * reordering is handled with a locked XOR (instead of a proper MFENCE
+ * instructon) for performance reasons. See PR 1674199 for more details.
+ *
+ * DMA barriers are equivalent to SMP barriers on x86.
+ *
+ * MMIO barriers are used to mix access to different memory types, so more
+ * reordering is possible, and is handled via LFENCE/SFENCE. Also, a proper
+ * MFENCE must be used instead of the locked XOR trick, due to the latter
+ * not guarding non-temporal/WC accesses.
+ */
+
+#define SMP_R_BARRIER_R()     COMPILER_READ_BARRIER()
+#define SMP_R_BARRIER_W()     COMPILER_MEM_BARRIER()
+#define SMP_R_BARRIER_RW()    COMPILER_MEM_BARRIER()
+#define SMP_W_BARRIER_R()     LOCKED_INSN_BARRIER()
+#define SMP_W_BARRIER_W()     COMPILER_WRITE_BARRIER()
+#define SMP_W_BARRIER_RW()    LOCKED_INSN_BARRIER()
+#define SMP_RW_BARRIER_R()    LOCKED_INSN_BARRIER()
+#define SMP_RW_BARRIER_W()    COMPILER_MEM_BARRIER()
+#define SMP_RW_BARRIER_RW()   LOCKED_INSN_BARRIER()
+
+/*
+ * Like the above, only for use with observers other than CPUs,
+ * i.e. DMA masters. Same as SMP barriers for x86.
+ */
+
+#define DMA_R_BARRIER_R()     SMP_R_BARRIER_R()
+#define DMA_R_BARRIER_W()     SMP_R_BARRIER_W()
+#define DMA_R_BARRIER_RW()    SMP_R_BARRIER_RW()
+#define DMA_W_BARRIER_R()     SMP_W_BARRIER_R()
+#define DMA_W_BARRIER_W()     SMP_W_BARRIER_W()
+#define DMA_W_BARRIER_RW()    SMP_W_BARRIER_RW()
+#define DMA_RW_BARRIER_R()    SMP_RW_BARRIER_R()
+#define DMA_RW_BARRIER_W()    SMP_RW_BARRIER_W()
+#define DMA_RW_BARRIER_RW()   SMP_RW_BARRIER_RW()
+
+/*
+ * And finally a set for use with MMIO accesses. These barriers must be stronger
+ * because they are used when mixing accesses to different memory types.
+ */
+
+#define MMIO_R_BARRIER_R()    LFENCE()
+#define MMIO_R_BARRIER_W()    SMP_R_BARRIER_W()
+#define MMIO_R_BARRIER_RW()   LFENCE()
+#define MMIO_W_BARRIER_R()    MFENCE()
+#define MMIO_W_BARRIER_W()    SFENCE()
+#define MMIO_W_BARRIER_RW()   MFENCE()
+#define MMIO_RW_BARRIER_R()   MFENCE()
+#define MMIO_RW_BARRIER_W()   SFENCE()
+#define MMIO_RW_BARRIER_RW()  MFENCE()
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MMIORead8 --
+ *
+ *      IO read from address "addr".
+ *
+ * Results:
+ *      8-bit value at given location.
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE uint8
+MMIORead8(const volatile void *addr)
+{
+   volatile uint8 *addr8 = (volatile uint8 *) addr;
+
+   return *addr8;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MMIOWrite8 --
+ *
+ *      IO write to address "addr".
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE void
+MMIOWrite8(volatile void *addr, // IN
+           uint8 val)           // IN
+{
+   volatile uint8 *addr8 = (volatile uint8 *) addr;
+
+   *addr8 = val;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MMIORead16 --
+ *
+ *      IO read from address "addr".
+ *
+ * Results:
+ *      16-bit value at given location.
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE uint16
+MMIORead16(const volatile void *addr)
+{
+   volatile uint16 *addr16 = (volatile uint16 *) addr;
+
+   return *addr16;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MMIOWrite16 --
+ *
+ *      IO write to address "addr".
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE void
+MMIOWrite16(volatile void *addr,  // IN
+            uint16 val)           // IN
+{
+   volatile uint16 *addr16 = (volatile uint16 *) addr;
+
+   *addr16 = val;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MMIORead32 --
+ *
+ *      IO read from address "addr".
+ *
+ * Results:
+ *      32-bit value at given location.
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE uint32
+MMIORead32(const volatile void *addr)
+{
+   volatile uint32 *addr32 = (volatile uint32 *) addr;
+
+   return *addr32;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MMIOWrite32 --
+ *
+ *      IO write to address "addr".
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE void
+MMIOWrite32(volatile void *addr, // OUT
+            uint32 val)
+{
+   volatile uint32 *addr32 = (volatile uint32 *) addr;
+
+   *addr32 = val;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MMIORead64 --
+ *
+ *      IO read from address "addr".
+ *
+ * Results:
+ *      64-bit value at given location.
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE uint64
+MMIORead64(const volatile void *addr)
+{
+   volatile uint64 *addr64 = (volatile uint64 *) addr;
+
+   return *addr64;
+}
+
+
+/*
+ *----------------------------------------------------------------------
+ *
+ * MMIOWrite64 --
+ *
+ *      IO write to address "addr".
+ *
+ *----------------------------------------------------------------------
+ */
+static INLINE void
+MMIOWrite64(volatile void *addr, // OUT
+            uint64 val)
+{
+   volatile uint64 *addr64 = (volatile uint64 *) addr;
+
+   *addr64 = val;
+}
+
+#endif // _VM_BASIC_ASM_X86_COMMON_H_

vmmon-only/include/vm_basic_defs.h

@@ -0,0 +1,815 @@
+/*********************************************************
+ * Copyright (C) 2003-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vm_basic_defs.h --
+ *
+ *	Standard macros for VMware source code.
+ */
+
+#ifndef _VM_BASIC_DEFS_H_
+#define _VM_BASIC_DEFS_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMKDRIVERS
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+#include "vm_basic_types.h" // For INLINE.
+
+/* Checks for FreeBSD, filtering out VMKERNEL. */
+#if !defined(VMKERNEL) && defined(__FreeBSD__)
+#define __IS_FREEBSD__ 1
+#else
+#define __IS_FREEBSD__ 0
+#endif
+#define __IS_FREEBSD_VER__(ver) (__IS_FREEBSD__ && __FreeBSD_version >= (ver))
+
+/*
+ * <stddef.h> provides definitions for:
+ *   NULL, offsetof
+ * References:
+ *   C90 7.17, C99 7.19, C11 7.19
+ */
+#if !defined(VMKERNEL)
+#  include <stddef.h>
+#else
+   /*
+    * Vmkernel's bogus __FreeBSD__ value causes gcc <stddef.h> to break.
+    * Work around by doing similar things. Bug 2116887 and 2229647.
+    */
+#  ifndef offsetof
+      /*
+       * We use the builtin offset for gcc/clang, except when we're running
+       * under the vmkernel's GDB macro preprocessor, since gdb doesn't
+       * understand __builtin_offsetof.
+       */
+#     if defined VMKERNEL_GDB_MACRO_BUILDER
+#        define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER)
+#     else
+#        define offsetof __builtin_offsetof
+#     endif
+#  endif
+
+#  ifndef NULL
+#     ifdef  __cplusplus
+#        define NULL    0
+#     else
+#        define NULL    ((void *)0)
+#     endif
+#  endif
+
+#endif  // VMKERNEL
+
+#if defined _WIN32 && defined USERLEVEL
+   #include <windows.h> // for Sleep() and LOWORD() etc.
+   #undef GetFreeSpace  // Unpollute preprocessor namespace.
+#endif
+
+
+/*
+ * Simple macros
+ */
+
+#define VMW_CONTAINER_OF(ptr, type, member) \
+   ((type *)((char *)(ptr) - offsetof(type, member)))
+
+#ifndef ARRAYSIZE
+#define ARRAYSIZE(a) (sizeof (a) / sizeof *(a))
+#endif
+
+#ifndef MIN
+#define MIN(_a, _b)   (((_a) < (_b)) ? (_a) : (_b))
+#endif
+
+/* The Solaris 9 cross-compiler complains about these not being used */
+#ifndef sun
+static INLINE int
+Min(int a, int b)
+{
+   return a < b ? a : b;
+}
+#endif
+
+#ifndef MAX
+#define MAX(_a, _b)   (((_a) > (_b)) ? (_a) : (_b))
+#endif
+
+#ifndef sun
+static INLINE int
+Max(int a, int b)
+{
+   return a > b ? a : b;
+}
+#endif
+
+#define VMW_CLAMP(x, min, max) \
+   ((x) < (min) ? (min) : ((x) > (max) ? (max) : (x)))
+
+#define ROUNDUP(x,y)		(((x) + (y) - 1) / (y) * (y))
+#define ROUNDDOWN(x,y)		((x) / (y) * (y))
+#define ROUNDUPBITS(x, bits)	(((uintptr_t) (x) + MASK(bits)) & ~MASK(bits))
+#define ROUNDDOWNBITS(x, bits)	((uintptr_t) (x) & ~MASK(bits))
+#define CEILING(x, y)		(((x) + (y) - 1) / (y))
+#if defined __APPLE__
+#include <machine/param.h>
+#undef MASK
+#endif
+
+/*
+ * The MASK macro behaves badly when given negative numbers or numbers larger
+ * than the highest order bit number (e.g. 32 on a 32-bit machine) as an
+ * argument. The range 0..31 is safe.
+ */
+
+#define MASK(n)		((1 << (n)) - 1)	    /* make an n-bit mask */
+#define MASK64(n)	((CONST64U(1) << (n)) - 1)  /* make an n-bit mask */
+/*
+ * MASKRANGE64 makes a bit vector starting at bit lo and ending at bit hi.  No
+ * checking for lo < hi is done.
+ */
+#define MASKRANGE64(hi, lo)      (MASK64((hi) - (lo) + 1) << (lo))
+
+/* SIGNEXT64 sign extends a n-bit value to 64-bits. */
+#define SIGNEXT64(val, n)       (((int64)(val) << (64 - (n))) >> (64 - (n)))
+
+#define DWORD_ALIGN(x)          ((((x) + 3) >> 2) << 2)
+#define QWORD_ALIGN(x)          ((((x) + 7) >> 3) << 3)
+
+#define IMPLIES(a,b) (!(a) || (b))
+
+
+/*
+ * Token concatenation
+ *
+ * The C preprocessor doesn't prescan arguments when they are
+ * concatenated or stringified.  So we need extra levels of
+ * indirection to convince the preprocessor to expand its
+ * arguments.
+ */
+
+#define CONC(x, y)              x##y
+#define XCONC(x, y)             CONC(x, y)
+#define XXCONC(x, y)            XCONC(x, y)
+#define MAKESTR(x)              #x
+#define XSTR(x)                 MAKESTR(x)
+
+
+/*
+ * Wide versions of string constants.
+ */
+
+#ifndef WSTR
+#define WSTR_(X)     L ## X
+#define WSTR(X)      WSTR_(X)
+#endif
+
+
+/*
+ * Page operations
+ *
+ * It has been suggested that these definitions belong elsewhere
+ * (like x86types.h).  However, I deem them common enough
+ * (since even regular user-level programs may want to do
+ * page-based memory manipulation) to be here.
+ * -- edward
+ */
+
+#ifndef PAGE_SHIFT // {
+#if defined __x86_64__ || defined __i386__
+   #define PAGE_SHIFT    12
+#elif defined __APPLE__
+   #define PAGE_SHIFT    12
+#elif defined VM_ARM_64
+   #define PAGE_SHIFT    12
+#elif defined __arm__
+   #define PAGE_SHIFT    12
+#else
+   #error
+#endif
+#endif // }
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE     (1 << PAGE_SHIFT)
+#endif
+
+#ifndef PAGE_MASK
+#define PAGE_MASK     (PAGE_SIZE - 1)
+#endif
+
+#ifndef PAGE_OFFSET
+#define PAGE_OFFSET(_addr)  ((uintptr_t)(_addr) & (PAGE_SIZE - 1))
+#endif
+
+#ifndef PAGE_NUMBER
+#define PAGE_NUMBER(_addr)  ((uintptr_t)(_addr) / PAGE_SIZE)
+#endif
+
+#ifndef VM_PAGE_BASE
+#define VM_PAGE_BASE(_addr)  ((_addr) & ~(PAGE_SIZE - 1))
+#endif
+
+#ifndef VM_PAGES_SPANNED
+#define VM_PAGES_SPANNED(_addr, _size) \
+   ((((_addr) & (PAGE_SIZE - 1)) + (_size) + (PAGE_SIZE - 1)) >> PAGE_SHIFT)
+#endif
+
+#ifndef BYTES_2_PAGES
+#define BYTES_2_PAGES(_nbytes)  ((_nbytes) >> PAGE_SHIFT)
+#endif
+
+#ifndef PAGES_2_BYTES
+#define PAGES_2_BYTES(_npages)  (((uint64)(_npages)) << PAGE_SHIFT)
+#endif
+
+#ifndef KBYTES_SHIFT
+#define KBYTES_SHIFT 10
+#endif
+
+#ifndef MBYTES_SHIFT
+#define MBYTES_SHIFT 20
+#endif
+
+#ifndef MBYTES_2_PAGES
+#define MBYTES_2_PAGES(_nbytes) \
+   ((uint64)(_nbytes) << (MBYTES_SHIFT - PAGE_SHIFT))
+#endif
+
+#ifndef PAGES_2_MBYTES
+#define PAGES_2_MBYTES(_npages) ((_npages) >> (MBYTES_SHIFT - PAGE_SHIFT))
+#endif
+
+#ifndef ROUNDUP_PAGES_2_MBYTES
+#define ROUNDUP_PAGES_2_MBYTES(_npages) \
+(((_npages) + MASK(MBYTES_SHIFT - PAGE_SHIFT)) >> (MBYTES_SHIFT - PAGE_SHIFT))
+#endif
+
+#ifndef ROUNDDOWN_PAGES_2_MBYTES
+#define ROUNDDOWN_PAGES_2_MBYTES(_npages) \
+((_npages) >> (MBYTES_SHIFT - PAGE_SHIFT))
+#endif
+
+#ifndef GBYTES_2_PAGES
+#define GBYTES_2_PAGES(_nbytes) ((uint64)(_nbytes) << (30 - PAGE_SHIFT))
+#endif
+
+#ifndef PAGES_2_GBYTES
+#define PAGES_2_GBYTES(_npages) ((_npages) >> (30 - PAGE_SHIFT))
+#endif
+
+#ifndef BYTES_2_KBYTES
+#define BYTES_2_KBYTES(_nbytes) ((_nbytes) >> KBYTES_SHIFT)
+#endif
+
+#ifndef KBYTES_2_BYTES
+#define KBYTES_2_BYTES(_nbytes) ((uint64)(_nbytes) << KBYTES_SHIFT)
+#endif
+
+#ifndef BYTES_2_MBYTES
+#define BYTES_2_MBYTES(_nbytes) ((_nbytes) >> MBYTES_SHIFT)
+#endif
+
+#ifndef MBYTES_2_BYTES
+#define MBYTES_2_BYTES(_nbytes) ((uint64)(_nbytes) << MBYTES_SHIFT)
+#endif
+
+#ifndef BYTES_2_GBYTES
+#define BYTES_2_GBYTES(_nbytes) ((_nbytes) >> 30)
+#endif
+
+#ifndef GBYTES_2_BYTES
+#define GBYTES_2_BYTES(_nbytes) ((uint64)(_nbytes) << 30)
+#endif
+
+#ifndef VM_PAE_LARGE_PAGE_SHIFT
+#define VM_PAE_LARGE_PAGE_SHIFT 21
+#endif
+
+#ifndef VM_PAE_LARGE_PAGE_SIZE
+#define VM_PAE_LARGE_PAGE_SIZE (1 << VM_PAE_LARGE_PAGE_SHIFT)
+#endif
+
+#ifndef VM_PAE_LARGE_PAGE_MASK
+#define VM_PAE_LARGE_PAGE_MASK (VM_PAE_LARGE_PAGE_SIZE - 1)
+#endif
+
+#ifndef VM_PAE_LARGE_2_SMALL_PAGES
+#define VM_PAE_LARGE_2_SMALL_PAGES (BYTES_2_PAGES(VM_PAE_LARGE_PAGE_SIZE))
+#endif
+
+#ifndef VM_PAE_LARGE_2_BYTES
+#define VM_PAE_LARGE_2_BYTES(_2mbytes) ((_2mbytes) << VM_PAE_LARGE_PAGE_SHIFT)
+#endif
+
+#ifndef VM_1GB_PAGE_SHIFT
+#define VM_1GB_PAGE_SHIFT 30
+#endif
+
+#ifndef VM_1GB_PAGE_SIZE
+#define VM_1GB_PAGE_SIZE (1 << VM_1GB_PAGE_SHIFT)
+#endif
+
+#ifndef VM_1GB_2_PAGES
+#define VM_1GB_2_PAGES (BYTES_2_PAGES(VM_1GB_PAGE_SIZE))
+#endif
+
+#ifndef VM_1GB_2_PDIRS
+#define VM_1GB_2_PDIRS (VM_1GB_PAGE_SIZE / VM_PAE_LARGE_PAGE_SIZE)
+#endif
+
+/*
+ * Word operations
+ */
+
+#ifndef LOWORD
+#define LOWORD(_dw)   ((_dw) & 0xffff)
+#endif
+#ifndef HIWORD
+#define HIWORD(_dw)   (((_dw) >> 16) & 0xffff)
+#endif
+
+#ifndef LOBYTE
+#define LOBYTE(_w)    ((_w) & 0xff)
+#endif
+#ifndef HIBYTE
+#define HIBYTE(_w)    (((_w) >> 8) & 0xff)
+#endif
+
+#ifndef HIDWORD
+#define HIDWORD(_qw)   ((uint32)((_qw) >> 32))
+#endif
+#ifndef LODWORD
+#define LODWORD(_qw)   ((uint32)(_qw))
+#endif
+#define QWORD(_hi, _lo)   ((((uint64)(_hi)) << 32) | ((uint32)(_lo)))
+
+
+/*
+ * Deposit a field _src at _pos bits from the right,
+ * with a length of _len, into the integer _target.
+ */
+
+#define DEPOSIT_BITS(_src,_pos,_len,_target) { \
+	unsigned mask = ((1 << _len) - 1); \
+	unsigned shiftedmask = ((1 << _len) - 1) << _pos; \
+	_target = (_target & ~shiftedmask) | ((_src & mask) << _pos); \
+}
+
+
+/*
+ * Get return address.
+ */
+
+#ifdef _MSC_VER
+#ifdef __cplusplus
+extern "C"
+#endif
+void *_ReturnAddress(void);
+#pragma intrinsic(_ReturnAddress)
+#define GetReturnAddress() _ReturnAddress()
+#elif __GNUC__
+#define GetReturnAddress() __builtin_return_address(0)
+#endif
+
+
+#ifdef __GNUC__
+#ifndef sun
+
+/*
+ * A bug in __builtin_frame_address was discovered in gcc 4.1.1, and
+ * fixed in 4.2.0; assume it originated in 4.0. PR 147638 and 554369.
+ */
+#if  !(__GNUC__ == 4 && (__GNUC_MINOR__ == 0 || __GNUC_MINOR__ == 1))
+#define GetFrameAddr() __builtin_frame_address(0)
+#endif
+
+#endif // sun
+#endif // __GNUC__
+
+/*
+ * Data prefetch was added in gcc 3.1.1
+ * http://www.gnu.org/software/gcc/gcc-3.1/changes.html
+ */
+#ifdef __GNUC__
+#  if ((__GNUC__ > 3) || (__GNUC__ == 3 && __GNUC_MINOR__ > 1) || \
+       (__GNUC__ == 3 && __GNUC_MINOR__ == 1 && __GNUC_PATCHLEVEL__ >= 1))
+#     define PREFETCH_R(var) __builtin_prefetch((var), 0 /* read */, \
+                                                3 /* high temporal locality */)
+#     define PREFETCH_W(var) __builtin_prefetch((var), 1 /* write */, \
+                                                3 /* high temporal locality */)
+#  else
+#     define PREFETCH_R(var) ((void)(var))
+#     define PREFETCH_W(var) ((void)(var))
+#  endif
+#endif /* __GNUC__ */
+
+
+#ifdef USERLEVEL // {
+
+/*
+ * Standardize some Posix names on Windows.
+ */
+
+#ifdef _WIN32 // {
+
+/* Conflict with definition of Visual Studio 2015 */
+#if (_MSC_VER < 1900)
+#define snprintf  _snprintf
+#endif
+
+#define strtok_r  strtok_s
+
+typedef int uid_t;
+typedef int gid_t;
+
+static INLINE void
+sleep(unsigned int sec)
+{
+   Sleep(sec * 1000);
+}
+
+static INLINE int
+usleep(unsigned long usec)
+{
+   Sleep(CEILING(usec, 1000));
+
+   return 0;
+}
+
+typedef int pid_t;
+#define       F_OK          0
+#define       X_OK          1
+#define       W_OK          2
+#define       R_OK          4
+
+#endif // } _WIN32
+
+#endif // } USERLEVEL
+
+#ifndef va_copy
+
+#ifdef _WIN32
+
+/*
+ * Windows needs va_copy. This works for both 32 and 64-bit Windows
+ * based on inspection of how varags.h from the Visual C CRTL is
+ * implemented. (Future versions of the RTL may break this).
+ */
+
+#define va_copy(dest, src) ((dest) = (src))
+
+#elif defined(__APPLE__) && defined(KERNEL)
+
+// The macOS kernel SDK defines va_copy in stdarg.h.
+#include <stdarg.h>
+
+#elif defined(__GNUC__) && (__GNUC__ < 3)
+
+/*
+ * Old versions of gcc recognize __va_copy, but not va_copy.
+ */
+
+#define va_copy(dest, src) __va_copy(dest, src)
+
+#endif // _WIN32
+
+#endif // va_copy
+
+/*
+ * This one is outside USERLEVEL because it's used by
+ * files compiled into the Windows hgfs driver or the display
+ * driver.
+ */
+
+#if defined(_WIN32) && defined(_MSC_VER)
+#define PATH_MAX 256
+#ifndef strcasecmp
+#define strcasecmp(_s1,_s2)   _stricmp((_s1),(_s2))
+#endif
+#ifndef strncasecmp
+#define strncasecmp(_s1,_s2,_n)   _strnicmp((_s1),(_s2),(_n))
+#endif
+#endif
+
+/*
+ * Convenience definitions of unicode characters.
+ */
+
+#ifndef UTF8_ELLIPSIS
+#define UTF8_ELLIPSIS "\xe2\x80\xa6"
+#endif
+
+/*
+ * Convenience macros and definitions. Can often be used instead of #ifdef.
+ */
+
+#undef ARM64_ONLY
+#ifdef VM_ARM_64
+#define ARM64_ONLY(x)    x
+#else
+#define ARM64_ONLY(x)
+#endif
+
+#undef X86_ONLY
+#ifdef VM_X86_ANY
+#define X86_ONLY(x)      x
+#else
+#define X86_ONLY(x)
+#endif
+
+#undef DEBUG_ONLY
+#ifdef VMX86_DEBUG
+#define vmx86_debug      1
+#define DEBUG_ONLY(...)  __VA_ARGS__
+#else
+#define vmx86_debug      0
+#define DEBUG_ONLY(...)
+#endif
+
+#ifdef VMX86_STATS
+#define vmx86_stats   1
+#define STATS_ONLY(x) x
+#else
+#define vmx86_stats   0
+#define STATS_ONLY(x)
+#endif
+
+#ifdef VMX86_DEVEL
+#define vmx86_devel   1
+#define DEVEL_ONLY(x) x
+#else
+#define vmx86_devel   0
+#define DEVEL_ONLY(x)
+#endif
+
+#ifdef VMX86_LOG
+#define vmx86_log     1
+#define LOG_ONLY(x)   x
+#else
+#define vmx86_log     0
+#define LOG_ONLY(x)
+#endif
+
+#ifdef VMX86_BETA
+#define vmx86_beta     1
+#define BETA_ONLY(x)   x
+#else
+#define vmx86_beta     0
+#define BETA_ONLY(x)
+#endif
+
+#ifdef VMX86_RELEASE
+#define vmx86_release   1
+#define RELEASE_ONLY(x) x
+#else
+#define vmx86_release   0
+#define RELEASE_ONLY(x)
+#endif
+
+#ifdef VMX86_SERVER
+#define vmx86_server 1
+#define SERVER_ONLY(x) x
+#define HOSTED_ONLY(x)
+#else
+#define vmx86_server 0
+#define SERVER_ONLY(x)
+#define HOSTED_ONLY(x) x
+#endif
+
+#ifdef VMKERNEL
+#define vmkernel 1
+#define VMKERNEL_ONLY(x) x
+#else
+#define vmkernel 0
+#define VMKERNEL_ONLY(x)
+#endif
+
+#ifdef _WIN32
+#define WIN32_ONLY(x) x
+#define POSIX_ONLY(x)
+#define vmx86_win32 1
+#else
+#define WIN32_ONLY(x)
+#define POSIX_ONLY(x) x
+#define vmx86_win32 0
+#endif
+
+#ifdef __linux__
+#define vmx86_linux 1
+#define LINUX_ONLY(x) x
+#else
+#define vmx86_linux 0
+#define LINUX_ONLY(x)
+#endif
+
+#ifdef __APPLE__
+#define vmx86_apple 1
+#define APPLE_ONLY(x) x
+#else
+#define vmx86_apple 0
+#define APPLE_ONLY(x)
+#endif
+
+#if defined(__APPLE__) && defined(VMW_APPLE_SANDBOX)
+#define vmw_apple_sandbox 1
+#else
+#define vmw_apple_sandbox 0
+#endif
+
+#if defined(__APPLE__) && defined(VMW_APPLE_APP_STORE)
+#define vmw_apple_app_store 1
+#else
+#define vmw_apple_app_store 0
+#endif
+
+#ifdef VMM
+#define vmx86_vmm 1
+#define VMM_ONLY(x) x
+#else
+#define vmx86_vmm 0
+#define VMM_ONLY(x)
+#endif
+
+#ifdef ULM
+#define vmx86_ulm 1
+#define ULM_ONLY(x) x
+#else
+#define vmx86_ulm 0
+#define ULM_ONLY(x)
+#endif
+
+#if defined(VMM) || defined(ULM)
+#define MONITOR_ONLY(x) x
+#else
+#define MONITOR_ONLY(x)
+#endif
+
+#if defined(VMM) || defined(VMKERNEL)
+#define USER_ONLY(x)
+#else
+#define USER_ONLY(x) x
+#endif
+
+#ifdef _WIN32
+#define VMW_INVALID_HANDLE INVALID_HANDLE_VALUE
+#else
+#define VMW_INVALID_HANDLE (-1LL)
+#endif
+
+#ifdef _WIN32
+#define fsync(fd) _commit(fd)
+#define fileno(f) _fileno(f)
+#else
+#endif
+
+/*
+ * Debug output macros for Windows drivers (the Eng variant is for
+ * display/printer drivers only.
+ */
+#ifdef _WIN32
+#if defined(VMX86_LOG)
+#ifdef _WIN64
+#define WinDrvPrint(arg, ...) DbgPrintEx(DPFLTR_IHVDRIVER_ID, (ULONG)~0, arg, __VA_ARGS__)
+#else
+#define WinDrvPrint(arg, ...) DbgPrint(arg, __VA_ARGS__)
+#endif
+#define WinDrvEngPrint(arg, ...) EngDbgPrint(arg, __VA_ARGS__)
+#else
+#define WinDrvPrint(arg, ...)
+#define WinDrvEngPrint(arg, ...)
+#endif
+#endif // _WIN32
+
+/*
+ * Use to initialize cbSize for this structure to preserve < Vista
+ * compatibility.
+ */
+#define NONCLIENTMETRICSINFO_V1_SIZE CCSIZEOF_STRUCT(NONCLIENTMETRICS, \
+                                                     lfMessageFont)
+
+/* This is not intended to be thread-safe. */
+#define DO_ONCE(code)                                                   \
+   do {                                                                 \
+      static MONITOR_ONLY(PERVCPU) Bool _doOnceDone = FALSE;            \
+      if (UNLIKELY(!_doOnceDone)) {                                     \
+         _doOnceDone = TRUE;                                            \
+         code;                                                          \
+      }                                                                 \
+   } while (0)
+
+/*
+ * Bug 827422 and 838523.
+ */
+
+#if defined __GNUC__ && __GNUC__ >= 4
+#define VISIBILITY_HIDDEN __attribute__((visibility("hidden")))
+#else
+#define VISIBILITY_HIDDEN /* nothing */
+#endif
+
+
+/*
+ * Bitfield extraction.
+ */
+
+#define EXTRACT_BITSLICE32(_val , _lsb, _msb)  \
+   (((uint32)(_val) << (31 - (_msb))) >> ((31 - (_msb)) + (_lsb)))
+#define EXTRACT_BITFIELD32(_val, _pos, _len) \
+   EXTRACT_BITSLICE32((_val), (_pos), ((_pos) + (_len) - 1))
+#define EXTRACT_BITSLICE64(_val, _lsb, _msb) \
+   (((uint64)(_val) << (63 - (_msb))) >> ((63 - (_msb)) + (_lsb)))
+#define EXTRACT_BITFIELD64(_val, _pos, _len) \
+   EXTRACT_BITSLICE64((_val), (_pos), ((_pos) + (_len) - 1))
+
+/*
+ * Typical cache line size.  Use this for aligning structures to cache
+ * lines for performance, but do not rely on it for correctness.
+ *
+ * On x86, all current processors newer than P4 have 64-byte lines,
+ * but P4 had 128.
+ *
+ * On ARM, the line size can vary between cores.  64-byte lines are
+ * common, but either larger or smaller powers of two are possible.
+ */
+#define CACHELINE_SIZE             64
+#define CACHELINE_SHIFT            6
+#define CACHELINE_ALIGNMENT_MASK   (CACHELINE_SIZE - 1)
+
+
+/*
+ * Bits to bytes sizes.
+ */
+
+#define SIZE_8BIT   1
+#define SIZE_16BIT  2
+#define SIZE_24BIT  3
+#define SIZE_32BIT  4
+#define SIZE_48BIT  6
+#define SIZE_64BIT  8
+#define SIZE_80BIT  10
+#define SIZE_128BIT 16
+#define SIZE_256BIT 32
+#define SIZE_512BIT 64
+
+/*
+ * Allocate a variable of type _type, aligned to _align bytes, returning a
+ * pointer to the variable in _var.  Potentially _align - 1 bytes may be
+ * wasted.  On x86, GCC 6.3.0 behaves sub-optimally when variables are declared
+ * on the stack using the aligned attribute, so this pattern is preferred.
+ * See PRs 1795155, 1819963.
+ */
+#define WITH_PTR_TO_ALIGNED_VAR(_type, _align, _var)                     \
+   do {                                                                  \
+      uint8 _buf_##_var[sizeof(_type) + (_align) - 1];                   \
+      _type *_var = (_type *) ((uintptr_t)(_buf_##_var + (_align) - 1) & \
+                               ~((uintptr_t) ((_align) - 1)));
+
+#define END_PTR_TO_ALIGNED_VAR \
+   } while (0)
+
+
+/*
+ * -Wswitch means that when you pass switch an enum that it's looking for
+ * all values from that enum, and only that enum, to be accounted for.
+ * "default:;" is fine for catching values you don't care about. But today
+ * we have a bunch of code that uses internal and external enum values, or
+ * in other words combines two enums into a single variable. This cast is
+ * the workaround, but we really need to fix this mess.
+ */
+#define UNCHECKED_SWITCH__FIXME(x) switch ((uint64)(x))
+
+
+/*
+ * When clang static analyzer parses source files, it implicitly defines
+ * __clang_analyzer__ macro. We use this to define our custom macro to stop
+ * its execution for the current path of analysis by calling a function that
+ * doesn't return, making it think that it hit a failed assertion.
+ *
+ * DO NOT use to silence the analyzer! See PR2447238.
+ */
+#ifdef __clang_analyzer__
+#define VMW_CLANG_ANALYZER_NORETURN() Panic("Disable Clang static analyzer")
+#else
+#define VMW_CLANG_ANALYZER_NORETURN() ((void)0)
+#endif
+
+#endif // ifndef _VM_BASIC_DEFS_H_

vmmon-only/include/vm_basic_math.h

@@ -0,0 +1,197 @@
+/*********************************************************
+ * Copyright (C) 2008-2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vm_basic_math.h --
+ *
+ *	Standard mathematical macros for VMware source code.
+ */
+
+#ifndef _VM_BASIC_MATH_H_
+#define _VM_BASIC_MATH_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMKDRIVERS
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+#include "vm_basic_types.h" // For INLINE.
+#include "vm_basic_asm.h"   // For Div64...
+
+#if defined __cplusplus
+extern "C" {
+#endif
+
+
+static INLINE uint32
+RatioOf(uint32 numer1, uint32 numer2, uint32 denom)
+{
+   uint64 numer = (uint64)numer1 * numer2;
+   /* Calculate "(numer1 * numer2) / denom" avoiding round-off errors. */
+#if defined(VMM) || !(defined(__i386__) || defined(__x86_64__))
+   return numer / denom;
+#else
+   uint32 ratio;
+   uint32 unused;
+   Div643232(numer, denom, &ratio, &unused);
+   return ratio;
+#endif
+}
+
+static INLINE uint32
+ExponentialAvg(uint32 avg, uint32 value, uint32 gainNumer, uint32 gainDenom)
+{
+   uint32 term1 = gainNumer * avg;
+   uint32 term2 = (gainDenom - gainNumer) * value;
+   return (term1 + term2) / gainDenom;
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * IsZeroOrPowerOfTwo --
+ * IsZeroOrPowerOfTwo64 --
+ *
+ * Results:
+ *      TRUE iff the value is 0 or a power of two.
+ *
+ * Side effects:
+ *      None.
+ *
+ *-----------------------------------------------------------------------------
+ */
+
+static INLINE Bool
+IsZeroOrPowerOfTwo64(uint64 x)
+{
+   return !(x & (x - 1));
+}
+
+
+static INLINE Bool
+IsZeroOrPowerOfTwo(uint32 x)     // IN
+{
+   return !(x & (x - 1));
+}
+
+
+static INLINE uint32
+GetPowerOfTwo(uint32 x)
+{
+   /* Returns next-greatest power-of-two. */
+   uint32 power2 = 1;
+   while (x > power2) {
+      power2 = power2 << 1;
+   }
+   return power2;
+}
+
+
+#if !defined(_WIN32) && !defined(_WIN64)
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * RotateLeft32 --
+ *
+ * Results:
+ *      Value rotated to the left by 'shift' bits.
+ *
+ * Side effects:
+ *      None.
+ *
+ *-----------------------------------------------------------------------------
+ */
+static INLINE uint32
+RotateLeft32(uint32 value, uint8 shift)
+{
+   return ((value << shift) | (value >> (32 - shift)));
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * RotateRight32 --
+ *
+ * Results:
+ *      Value rotated to the right by 'shift' bits.
+ *
+ * Side effects:
+ *      None.
+ *
+ *-----------------------------------------------------------------------------
+ */
+static INLINE uint32
+RotateRight32(uint32 value, uint8 shift)
+{
+   return ((value >> shift) | (value << (32 - shift)));
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * RotateLeft64 --
+ *
+ * Results:
+ *      Value rotated to the left by 'shift' bits.
+ *
+ * Side effects:
+ *      None.
+ *
+ *-----------------------------------------------------------------------------
+ */
+static INLINE uint64
+RotateLeft64(uint64 value, uint8 shift)
+{
+   return ((value << shift) | (value >> (64 - shift)));
+}
+
+
+/*
+ *-----------------------------------------------------------------------------
+ *
+ * RotateRight64 --
+ *
+ * Results:
+ *      Value rotated to the right by 'shift' bits.
+ *
+ * Side effects:
+ *      None.
+ *
+ *-----------------------------------------------------------------------------
+ */
+static INLINE uint64
+RotateRight64(uint64 value, uint8 shift)
+{
+   return ((value >> shift) | (value << (64 - shift)));
+}
+#endif // if !defined(_WIN32) && !defined(_WIN64)
+
+
+#if defined __cplusplus
+} // extern "C"
+#endif
+
+#endif // ifndef _VM_BASIC_MATH_H_

vmmon-only/include/vm_pagetable.h

@@ -0,0 +1,140 @@
+/*********************************************************
+ * Copyright (C) 2014-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vm_pagetable.h --
+ *
+ *      Virtual memory page table definitions
+ */
+
+#ifndef _VM_PAGETABLE_H_
+#define _VM_PAGETABLE_H_
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_USERLEVEL
+#include "includeCheck.h"
+
+#include "vm_basic_types.h"
+#include "vm_basic_defs.h"
+
+/*
+ * These definitions suit both the x64 and arm64 architectures. In particular
+ * - At all levels, a page table is a 4KB page containing 512 8-byte entries.
+ * - Each entry maps 48, 39, 30, 21 and 12 bits at the respective page table
+ *   levels (with standard page sizes).
+ */
+
+typedef uint64 PT_Entry;
+typedef PT_Entry PT_L5E;
+typedef PT_Entry PT_L4E;
+typedef PT_Entry PT_L3E;
+typedef PT_Entry PT_L2E;
+typedef PT_Entry PT_L1E;
+
+typedef enum {
+   PT_LEVEL_1 = 1,
+   PT_LEVEL_2,
+   PT_LEVEL_3,
+   PT_LEVEL_4,
+   PT_LEVEL_5,
+   PT_LEVEL_STOP = PT_LEVEL_1,
+   PT_MAX_LEVELS = PT_LEVEL_5
+} PT_Level;
+
+#define PT_PTE_SIZE           8
+#define PT_LEVEL_SHIFT        9
+
+#define PT_PTE_PFN_SHIFT      PAGE_SHIFT
+#define PT_ENTRIES_PER_PT     (1 << PT_LEVEL_SHIFT)
+#define PT_OFF_MASK           (PT_ENTRIES_PER_PT - 1)
+
+/* Align any page number type to the 2M containing page number. */
+#define PT_2M_ALIGN_PN(x)     ((x) & ~PT_OFF_MASK)
+
+/*
+ * log2 of the
+ * address space size (in 4KB pages) covered by a level-_l page table entry.
+ */
+#define PT_LE_PG_SHIFT(_l)    (PT_LEVEL_SHIFT * ((_l) - PT_LEVEL_1))
+
+/* Address space size (in 4KB pages) covered by a level-_l page table entry. */
+#define PT_PAGES_PER_LE(_l)   ((uint64)1 << PT_LE_PG_SHIFT(_l))
+#define PT_PAGES_PER_L5E      PT_PAGES_PER_LE(PT_LEVEL_5)
+#define PT_PAGES_PER_L4E      PT_PAGES_PER_LE(PT_LEVEL_4)
+#define PT_PAGES_PER_L3E      PT_PAGES_PER_LE(PT_LEVEL_3)
+#define PT_PAGES_PER_L2E      PT_PAGES_PER_LE(PT_LEVEL_2)
+#define PT_PAGES_PER_L1E      PT_PAGES_PER_LE(PT_LEVEL_1)
+
+/*
+ * log2 of the
+ * address space size (in bytes) covered by a level-_l page table entry.
+ */
+#define PT_LE_SHIFT(_l)       (PAGE_SHIFT + PT_LE_PG_SHIFT(_l))
+#define PT_L5E_SHIFT          PT_LE_SHIFT(PT_LEVEL_5)
+#define PT_L4E_SHIFT          PT_LE_SHIFT(PT_LEVEL_4)
+#define PT_L3E_SHIFT          PT_LE_SHIFT(PT_LEVEL_3)
+#define PT_L2E_SHIFT          PT_LE_SHIFT(PT_LEVEL_2)
+#define PT_L1E_SHIFT          PT_LE_SHIFT(PT_LEVEL_1)
+
+/* Address space size (in bytes) covered by a level-_l page table entry. */
+#define PT_LE_SIZE(_l)        ((uint64)1 << PT_LE_SHIFT(_l))
+#define PT_L5E_SIZE           PT_LE_SIZE(PT_LEVEL_5)
+#define PT_L4E_SIZE           PT_LE_SIZE(PT_LEVEL_4)
+#define PT_L3E_SIZE           PT_LE_SIZE(PT_LEVEL_3)
+#define PT_L2E_SIZE           PT_LE_SIZE(PT_LEVEL_2)
+#define PT_L1E_SIZE           PT_LE_SIZE(PT_LEVEL_1)
+
+/*
+ * For a given LA,
+ * determine the offset (== index) of the level-_l page table entry.
+ */
+#define PT_LA_2_LOFF(_a, _l)  (((_a) >> PT_LE_SHIFT(_l)) & PT_OFF_MASK)
+#define PT_LA_2_L5OFF(_a)     PT_LA_2_LOFF(_a, PT_LEVEL_5)
+#define PT_LA_2_L4OFF(_a)     PT_LA_2_LOFF(_a, PT_LEVEL_4)
+#define PT_LA_2_L3OFF(_a)     PT_LA_2_LOFF(_a, PT_LEVEL_3)
+#define PT_LA_2_L2OFF(_a)     PT_LA_2_LOFF(_a, PT_LEVEL_2)
+#define PT_LA_2_L1OFF(_a)     PT_LA_2_LOFF(_a, PT_LEVEL_1)
+
+/*
+ * For a given LPN,
+ * determine the offset (== index) of the level-_l page table entry.
+ */
+#define PT_LPN_2_LOFF(_a, _l) (((_a) >> PT_LE_PG_SHIFT(_l)) & PT_OFF_MASK)
+#define PT_LPN_2_L5OFF(_a)    PT_LPN_2_LOFF(_a, PT_LEVEL_5)
+#define PT_LPN_2_L4OFF(_a)    PT_LPN_2_LOFF(_a, PT_LEVEL_4)
+#define PT_LPN_2_L3OFF(_a)    PT_LPN_2_LOFF(_a, PT_LEVEL_3)
+#define PT_LPN_2_L2OFF(_a)    PT_LPN_2_LOFF(_a, PT_LEVEL_2)
+#define PT_LPN_2_L1OFF(_a)    PT_LPN_2_LOFF(_a, PT_LEVEL_1)
+
+/* Size (in bytes) of a 1GB page. */
+#define PT_1G_PAGE_SIZE       PT_L3E_SIZE
+#define PT_1G_PAGE_MASK       (PT_L3E_SIZE - 1)
+
+/* Size (in bytes) of a 2MB page. */
+#define PT_2M_PAGE_SIZE       PT_L2E_SIZE
+#define PT_2M_PAGE_MASK       (PT_L2E_SIZE - 1)
+
+#define PT_LE_NXT_PG(_a, _l)  ((((_a) >> PT_LE_PG_SHIFT(_l)) + 1) \
+                               << PT_LE_PG_SHIFT(_l))
+
+#endif

vmmon-only/include/vm_time.h

@@ -0,0 +1,55 @@
+/*********************************************************
+ * Copyright (C) 1998 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+
+/*
+ * vm_time.h  --
+ *
+ *    Time management functions.
+ *    Part of driver-only distribution
+ *
+ *    see comment in poll.c
+ */
+
+
+#ifndef VM_TIME_H
+#define VM_TIME_H
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+
+struct VmTimeVirtualRealClock;
+typedef struct VmTimeVirtualRealClock VmTimeVirtualRealClock;
+
+#ifdef USERLEVEL
+
+VmTimeType VmTime_ReadVirtualTime(void);
+VmTimeVirtualRealClock *VmTime_NewVirtualRealClock(void);
+void VmTime_StartVirtualRealClock(VmTimeVirtualRealClock *, double);
+void VmTime_ResetVirtualRealClock(VmTimeVirtualRealClock *);
+VmTimeType VmTime_ReadVirtualRealTime(VmTimeVirtualRealClock *);
+VmTimeType VmTime_RemainingVirtualRealTime(VmTimeVirtualRealClock *,
+                                           VmTimeType realTime);
+void VmTime_UpdateVirtualRealTime(VmTimeVirtualRealClock *clock,
+                                  VmTimeType realTime,
+                                  VmTimeType virtualTime);
+#endif
+#endif /* VM_TIME_H */
+

vmmon-only/include/vmm_constants.h

@@ -0,0 +1,42 @@
+/*********************************************************
+ * Copyright (C) 1998-2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/* Do not include inline functions, typedefs or other such
+ * nonsense here.   This file is for _constants_.
+ */
+
+#ifndef _VMM_CONSTANTS_H_
+#define _VMM_CONSTANTS_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+
+#define VMM_PANIC_NONE         0
+#define VMM_PANIC_MSG_COPYING  1
+#define VMM_PANIC_MSG_COPIED   2
+#define VMM_PANIC_COREDUMPING  3
+#define VMM_PANIC_VCPU         4
+
+#define VMM_PANIC_MSG_SIZE     256
+
+#endif

vmmon-only/include/vmmblob.h

@@ -0,0 +1,50 @@
+/*********************************************************
+ * Copyright (C) 2017-2018 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vmmblob.h --
+ *
+ *     VMM blob management.
+ */
+
+#ifndef VMMBLOB_H
+#define VMMBLOB_H
+
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#include "includeCheck.h"
+
+struct VMDriver;
+struct MonLoaderHeader;
+
+typedef struct VmmBlobInfo {
+   uint8                  *blobPtr;
+   uint32                  numBytes;
+   MPN                    *mpns;
+   struct MonLoaderHeader *header;
+} VmmBlobInfo;
+
+void   VmmBlob_Cleanup(VmmBlobInfo *bi);
+Bool   VmmBlob_Load(VA64 bsBlobAddr, uint32 numBytes, uint32 headerOffset,
+                    VmmBlobInfo **blobInfo);
+MPN    VmmBlob_GetMpn(struct VMDriver *vm, uint64 blobOffset);
+MPN    VmmBlob_GetHeaderMpn(struct VMDriver *vm);
+uint8 *VmmBlob_GetPtr(VMDriver *vm);
+uint64 VmmBlob_GetSize(VMDriver *vm);
+
+#endif /* VMMBLOB_H */

vmmon-only/include/vmmem_shared.h

@@ -0,0 +1,116 @@
+/*********************************************************
+ * Copyright (C) 2000-2015,2017-2020 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vmmem_shared.h --
+ *
+ *      This is the header file for machine memory manager.
+ */
+
+
+#ifndef _VMMEM_SHARED_H
+#define _VMMEM_SHARED_H
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMX
+#include "includeCheck.h"
+
+/*
+ * Page remapping definitions.
+ */
+
+#define VMMEM_FLAG_BIT(x) (1 << (x))
+
+#define VMMEM_ANON_LOW_MEM        VMMEM_FLAG_BIT(0)
+#define VMMEM_ANON_CONTIG         VMMEM_FLAG_BIT(1)
+#define VMMEM_ANON_CAN_FAIL       VMMEM_FLAG_BIT(2)
+#define VMMEM_ANON_USE_PREALLOC   VMMEM_FLAG_BIT(3)
+#define VMMEM_ANON_IOABLE_PAGE    VMMEM_FLAG_BIT(4)
+#define VMMEM_ANON_ALL_FLAGS      MASK(5)
+
+#define VMMEM_GUEST_NO_FLAGS      0
+#define VMMEM_GUEST_WRITEABLE     VMMEM_FLAG_BIT(0)
+#define VMMEM_GUEST_BREAKCOW      VMMEM_FLAG_BIT(1)
+#define VMMEM_GUEST_2M_PAGE       VMMEM_FLAG_BIT(2)
+#define VMMEM_GUEST_1G_PAGE       VMMEM_FLAG_BIT(3)
+#define VMMEM_GUEST_CAN_FAIL      VMMEM_FLAG_BIT(4)
+#define VMMEM_GUEST_TEST_ZEROCOW  VMMEM_FLAG_BIT(5)
+#define VMMEM_GUEST_TRY_ZEROCOW   VMMEM_FLAG_BIT(6)
+#define VMMEM_GUEST_TRY_POISONCOW VMMEM_FLAG_BIT(7)
+#define VMMEM_GUEST_PREALLOC      VMMEM_FLAG_BIT(8)
+#define VMMEM_GUEST_ALL_FLAGS     MASK(9)
+#define VMMEM_GUEST_TRY_COW       (VMMEM_GUEST_TEST_ZEROCOW | \
+                                   VMMEM_GUEST_TRY_ZEROCOW  | \
+                                   VMMEM_GUEST_TRY_POISONCOW)
+#define VMMEM_GUEST_LARGE_PAGE    (VMMEM_GUEST_2M_PAGE | \
+                                   VMMEM_GUEST_1G_PAGE)
+
+#define VMMEM_PLATFORM_CHECK_OK           VMMEM_FLAG_BIT(0)
+#define VMMEM_PLATFORM_KEY_OK             VMMEM_FLAG_BIT(1)
+#define VMMEM_PLATFORM_COW                VMMEM_FLAG_BIT(2)
+#define VMMEM_PLATFORM_EXPOSED_TO_VMM     VMMEM_FLAG_BIT(3)
+#define VMMEM_PLATFORM_P2M_UPDATE_PENDING VMMEM_FLAG_BIT(4)
+#define VMMEM_PLATFORM_DIRTY              VMMEM_FLAG_BIT(5)
+#define VMMEM_PLATFORM_IS_2M_PAGE         VMMEM_FLAG_BIT(6)
+#define VMMEM_PLATFORM_IS_1G_PAGE         VMMEM_FLAG_BIT(7)
+#define VMMEM_PLATFORM_LARGE_RETRY        VMMEM_FLAG_BIT(8)
+#define VMMEM_PLATFORM_TRY_COW_SUCCESS    VMMEM_FLAG_BIT(9)
+
+#define VMMEM_PLATFORM_BACKED_LARGE      (VMMEM_PLATFORM_IS_2M_PAGE |   \
+                                          VMMEM_PLATFORM_IS_1G_PAGE)
+
+
+#define MAX_PLATFORM_PAGE_INFO_PAGES  240
+
+/*
+ * Structure used to query platform about the page state.
+ */
+typedef struct PlatformPageInfoList {
+   PageCnt numPages;
+   BPN     bpn[MAX_PLATFORM_PAGE_INFO_PAGES];    // bpns to check
+   MPN     mpn[MAX_PLATFORM_PAGE_INFO_PAGES];    // filled in by host
+   uint8   flags[MAX_PLATFORM_PAGE_INFO_PAGES];  // filled in by host
+} PlatformPageInfoList;
+
+#define VMMEM_SERVICES_TYPE_2_MASK(type)                  \
+           (1 << type)
+#define VMMEM_SERVICES_IN_MASK(typeMask, type)            \
+           (VMMEM_SERVICES_TYPE_2_MASK(type) & typeMask)
+#define VMMEM_SERVICES_CLEAR_MASK(typeMask, type)         \
+           (typeMask & ~VMMEM_SERVICES_TYPE_2_MASK(type))
+
+#define VMMEM_SERVICES_DEFS                                       \
+   MDEF(VMMEM_SERVICES_TYPE_P2M,         P2MUpdate_FilterPages)   \
+   MDEF(VMMEM_SERVICES_TYPE_SWAP,        BusMemSwap_FilterPages)  \
+   MDEF(VMMEM_SERVICES_TYPE_DIRTY_CLEAR, DirtyDrainer_ClearDirty)
+
+#define MDEF(_type, _cb) _type,
+typedef enum VmMemServices_Type {
+   VMMEM_SERVICES_DEFS
+#undef MDEF
+   VMMEM_SERVICES_TYPE_MAX
+} VmMemServices_Type;
+#define VMMEM_SERVICES_TYPE_INVALID (VMMEM_SERVICES_TYPE_MAX)
+
+void VmMem_DisableLargePageAllocations(void);
+void VmMem_EnableLargePageAllocations(void);
+
+#endif

vmmon-only/include/vmware.h

@@ -0,0 +1,57 @@
+/*********************************************************
+ * Copyright (C) 2003-2016 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vmware.h --
+ *
+ *	Standard include file for VMware source code.
+ */
+
+#ifndef _VMWARE_H_
+#define _VMWARE_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+#define INCLUDE_ALLOW_VMCORE
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_DISTRIBUTE
+#include "includeCheck.h"
+
+#include "vm_basic_types.h"
+#include "vm_basic_defs.h"
+#include "vm_assert.h"
+
+/*
+ * Global error codes. Currently used internally, but may be exported
+ * to customers one day, like VM_E_XXX in vmcontrol_constants.h
+ */
+
+typedef enum VMwareStatus {
+   VMWARE_STATUS_SUCCESS,  /* success */
+   VMWARE_STATUS_ERROR,    /* generic error */
+   VMWARE_STATUS_NOMEM,    /* generic memory allocation error */
+   VMWARE_STATUS_INSUFFICIENT_RESOURCES, /* internal or system resource limit exceeded */
+   VMWARE_STATUS_INVALID_ARGS  /* invalid arguments */
+} VMwareStatus;
+
+#define VMWARE_SUCCESS(s) ((s) == VMWARE_STATUS_SUCCESS)
+
+
+#endif // ifndef _VMWARE_H_

vmmon-only/include/vmware_pack_begin.h

@@ -0,0 +1,43 @@
+/*********************************************************
+ * Copyright (C) 2002-2016 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vmware_pack_begin.h --
+ *
+ *    Begin of structure packing. See vmware_pack_init.h for details.
+ *
+ *    Note that we do not use the following construct in this include file,
+ *    because we want to emit the code every time the file is included --hpreg
+ *
+ *    #ifndef foo
+ *    #   define foo
+ *    ...
+ *    #endif
+ *
+ */
+
+
+#include "vmware_pack_init.h"
+
+
+#ifdef _MSC_VER
+#   pragma pack(push, 1)
+#elif __GNUC__
+#else
+#   error Compiler packing...
+#endif

vmmon-only/include/vmware_pack_end.h

@@ -0,0 +1,44 @@
+/*********************************************************
+ * Copyright (C) 2002-2016 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * vmware_pack_end.h --
+ *
+ *    End of structure packing. See vmware_pack_init.h for details.
+ *
+ *    Note that we do not use the following construct in this include file,
+ *    because we want to emit the code every time the file is included --hpreg
+ *
+ *    #ifndef foo
+ *    #   define foo
+ *    ...
+ *    #endif
+ *
+ */
+
+
+#include "vmware_pack_init.h"
+
+
+#ifdef _MSC_VER
+#   pragma pack(pop)
+#elif __GNUC__
+__attribute__((__packed__))
+#else
+#   error Compiler packing...
+#endif

vmmon-only/include/vmware_pack_init.h

@@ -0,0 +1,65 @@
+/*********************************************************
+ * Copyright (C) 2002-2016 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef __VMWARE_PACK_INIT_H__
+#   define __VMWARE_PACK_INIT_H__
+
+
+/*
+ * vmware_pack_init.h --
+ *
+ *    Platform-independent code to make the compiler pack (i.e. have them
+ *    occupy the smallest possible space) structure definitions. The following
+ *    constructs are known to work --hpreg
+ *
+ *    #include "vmware_pack_begin.h"
+ *    struct foo {
+ *       ...
+ *    }
+ *    #include "vmware_pack_end.h"
+ *    ;
+ *
+ *    typedef
+ *    #include "vmware_pack_begin.h"
+ *    struct foo {
+ *       ...
+ *    }
+ *    #include "vmware_pack_end.h"
+ *    foo;
+ */
+
+
+#ifdef _MSC_VER
+/*
+ * MSVC 6.0 emits warning 4103 when the pack push and pop pragma pairing is
+ * not balanced within 1 included file. That is annoying because our scheme
+ * is based on the pairing being balanced between 2 included files.
+ *
+ * So we disable this warning, but this is safe because the compiler will also
+ * emit warning 4161 when there is more pops than pushes within 1 main
+ * file --hpreg
+ */
+
+#   pragma warning(disable:4103)
+#elif __GNUC__
+#else
+#   error Compiler packing...
+#endif
+
+
+#endif /* __VMWARE_PACK_INIT_H__ */

vmmon-only/include/x86_basic_defs.h

@@ -0,0 +1,276 @@
+/*********************************************************
+ * Copyright (C) 2006-2019 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+/*
+ * x86_basic_defs.h --
+ *
+ *    Basic macros describing the x86 architecture.
+ */
+
+#ifndef _X86_BASIC_DEFS_H_
+#define _X86_BASIC_DEFS_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_MODULE
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+#include <asm/processor-flags.h>
+
+#define X86_MAX_INSTR_LEN  15   /* Max byte length of an x86 instruction. */
+
+#define NUM_IDT_VECTORS 256
+
+/*
+ *   control registers
+ */
+
+#define CR0_PE         0x00000001
+#define CR0_MP         0x00000002
+#define CR0_EM         0x00000004
+#define CR0_TS         0x00000008
+#define CR0_ET         0x00000010
+#define CR0_NE         0x00000020
+#define CR0_WP         0x00010000
+#define CR0_AM         0x00040000
+#define CR0_NW         0x20000000
+#define CR0_CD         0x40000000
+#define CR0_PG         0x80000000
+
+#define CR0_CACHE_CONTROL (CR0_CD | CR0_NW)
+
+#define CR0_RESERVED   CONST64U(0xffffffff1ffaffc0)
+/*
+ * Note: The "Not Reserved" bits in CR0 are:
+ *   PG, CD, NW, AM, WP, NE, ET, TS, EM, MP, PE
+ *        |   |   |               |   |   |
+ *        |   |   +---------------+---+---+---> CR0_MUTABLE
+ *        |   |
+ *        +---+--> CR0_CACHE_CONTROL
+ *
+ * (CR0_MUTABLE is defined in vmkernel/private/x86/cpu.h)
+ */
+
+#define CR3_PWT        0x00000008
+#define CR3_PCD        0x00000010
+#define CR3_PDB_SHIFT  12
+#define CR3_PDB_MASK   0xfffff000
+#define CR3_IGNORE     0xFFF
+#define PAE_CR3_IGNORE 0x1F
+#ifndef CR3_PCID_MASK
+#define CR3_PCID_MASK  0xFFF
+#endif
+#define CR3_NO_FLUSH   (1ULL << 63)
+
+#define CR4_VME        0x00000001
+#define CR4_PVI        0x00000002
+#define CR4_TSD        0x00000004
+#define CR4_DE         0x00000008
+#define CR4_PSE        0x00000010
+#define CR4_PAE        0x00000020
+#define CR4_MCE        0x00000040
+#define CR4_PGE        0x00000080
+#define CR4_PCE        0x00000100
+#define CR4_OSFXSR     0x00000200 // CPU/OS supports SIMD insts
+#define CR4_OSXMMEXCPT 0x00000400 // #XF exception enable PIII only
+#define CR4_UMIP       0x00000800
+#define CR4_LA57       0x00001000
+#define CR4_VMXE       0x00002000
+#define CR4_SMXE       0x00004000
+#define CR4_FSGSBASE   0x00010000
+#define CR4_PCIDE      0x00020000
+#define CR4_OSXSAVE    0x00040000
+#define CR4_SMEP       0x00100000
+#define CR4_SMAP       0x00200000
+#define CR4_PKE        0x00400000
+#define CR4_RESERVED   CONST64U(0xffffffffff889000)
+#define CR8_RESERVED   CONST64U(0xfffffffffffffff0)
+
+/*
+ * Debug registers.
+ */
+
+#define DR_COUNT       4
+
+#define DR6_B0         0x00000001
+#define DR6_B1         0x00000002
+#define DR6_B2         0x00000004
+#define DR6_B3         0x00000008
+#define DR6_B0123      (DR6_B0 | DR6_B1 | DR6_B2 | DR6_B3)
+#define DR6_B(_n)      (1 << (_n))
+#define DR6_BD         0x00002000
+#define DR6_BS         0x00004000
+#define DR6_BT         0x00008000
+#define DR6_RTM        0x00010000
+#define DR6_ONES       0xfffe0ff0
+#define DR6_DEFAULT    (DR6_ONES | DR6_RTM)
+#define DR6_RESERVED_MASK 0xfffe1ff0
+
+#define DR7_L_MASK(_n)   (1 << ((_n) * 2))
+#define DR7_G_MASK(_n)   (1 << ((_n) * 2 + 1))
+#define DR7_LG_MASK(_n)  (3 << ((_n) * 2))
+#define DR7_RW_MASK(_n)  (3 << (16 + (_n) * 4))
+#define DR7_LEN_MASK(_n) (3 << (18 + (_n) * 4))
+#define DR7_BP_MASK(_n) (DR7_L_MASK(_n)  |\
+                         DR7_G_MASK(_n)  |\
+                         DR7_RW_MASK(_n) |\
+                         DR7_LEN_MASK(_n))
+
+#define DR7_L0         DR7_L_MASK(0)
+#define DR7_G0         DR7_G_MASK(0)
+#define DR7_L1         DR7_L_MASK(1)
+#define DR7_G1         DR7_G_MASK(1)
+#define DR7_L2         DR7_L_MASK(2)
+#define DR7_G2         DR7_G_MASK(2)
+#define DR7_L3         DR7_L_MASK(3)
+#define DR7_G3         DR7_G_MASK(3)
+#define DR7_ENABLED    0x000000ff
+
+#define DR7_LE         0x00000100    // Deprecated in modern hardware
+#define DR7_GE         0x00000200    // Deprecated in modern hardware
+#define DR7_GD         0x00002000
+#define DR7_ONES       0x00000400
+#define DR7_RTM        0x00000800
+#define DR7_RESERVED   CONST64U(0xffffffff0000d400)
+#define DR7_DEFUNCT    (DR7_LE | DR7_GE)
+#define DR7_DEFAULT    DR7_ONES
+#define DR7_LX_MASK    (DR7_L0 | DR7_L1 | DR7_L2 | DR7_L3 | DR7_LE)
+#define DR7_GX_MASK    (DR7_G0 | DR7_G1 | DR7_G2 | DR7_G3 | DR7_GE)
+#define DR7_LGX_MASK   (DR7_LX_MASK | DR7_GX_MASK)
+
+#define DR7_RW(_r,_n)  (((_r) >> (16+(_n)*4)) & 0x3)
+#define DR7_L(_r,_n)   (((_r) >> ((_n)*2)) & 1)
+#define DR7_G(_r,_n)   (((_r) >> (1 + (_n)*2)) & 1)
+#define DR7_LEN(_r,_n) (((_r) >> (18+(_n)*4)) & 0x3)
+
+#define DR7_RW_BITS(_n,_rw)     ((_rw) << (16 + (_n) * 4))
+#define DR7_LEN_BITS(_n,_len)   ((_len) << (18 + (_n) * 4))
+
+#define DR7_RW_INST    0x0
+#define DR7_RW_WRITES  0x1
+#define DR7_RW_IO      0x2
+#define DR7_RW_ACCESS  0x3
+
+#define DR7_LENGTH_1   0x0
+#define DR7_LENGTH_2   0x1
+#define DR7_LENGTH_8   0x2
+#define DR7_LENGTH_4   0x3
+
+#define DEBUG_STATUS_B0   (1<<0)
+#define DEBUG_STATUS_B1   (1<<1)
+#define DEBUG_STATUS_B2   (1<<2)
+#define DEBUG_STATUS_B3   (1<<3)
+#define DEBUG_STATUS_DB   (1<<13)
+#define DEBUG_STATUS_BS   (1<<14)
+#define DEBUG_STATUS_BT   (1<<15)
+
+/*
+ *   exception error codes
+ */
+
+#define EXC_DE            0
+#define EXC_DB            1
+#define EXC_NMI           2
+#define EXC_BP            3
+#define EXC_OF            4
+#define EXC_BR            5
+#define EXC_UD            6
+#define EXC_NM            7
+#define EXC_DF            8
+#define EXC_TS           10
+#define EXC_NP           11
+#define EXC_SS           12
+#define EXC_GP           13
+#define EXC_PF           14
+#define EXC_MF           16
+#define EXC_AC           17
+#define EXC_MC           18
+#define EXC_XF           19  // SIMD exception.
+#define EXC_VE           20  // Virtualization exception - VT only.
+#define EXC_VC           29  // VMM communication exception (SVM / SEV-ES only).
+#define EXC_SX           30  // Security exception (SVM only).
+
+/* Bitmap of the exception vectors that have associated error codes. */
+#define EXC_WITH_ERR_CODE_MASK ((1u << EXC_DF) | (1u << EXC_TS) | \
+                                (1u << EXC_NP) | (1u << EXC_SS) | \
+                                (1u << EXC_GP) | (1u << EXC_PF) | \
+                                (1u << EXC_AC))
+
+/*
+ * eflag/rflag definitions.
+ */
+
+#define EFLAGS_IOPL_SHIFT 12
+
+typedef enum x86_FLAGS {
+   EFLAGS_NONE         = 0,
+   EFLAGS_CF           = (1 << 0),     /* User */ 
+   EFLAGS_SET          = (1 << 1),                                  
+   EFLAGS_PF           = (1 << 2),     /* User */ 
+   EFLAGS_AF           = (1 << 4),     /* User */ 
+   EFLAGS_ZF           = (1 << 6),     /* User */ 
+   EFLAGS_SF           = (1 << 7),     /* User */ 
+   EFLAGS_TF           = (1 << 8),     /* Priv */ 
+   EFLAGS_IF           = (1 << 9),     /* Priv */ 
+   EFLAGS_DF           = (1 << 10),    /* User */ 
+   EFLAGS_OF           = (1 << 11),    /* User */ 
+   EFLAGS_NT           = (1 << 14),    /* Priv */ 
+   EFLAGS_RF           = (1 << 16),    /* Priv */ 
+   EFLAGS_VM           = (1 << 17),    /* Priv */ 
+   EFLAGS_AC           = (1 << 18),    /* Priv */ 
+   EFLAGS_VIF          = (1 << 19),    /* Priv */ 
+   EFLAGS_VIP          = (1 << 20),    /* Priv */ 
+   EFLAGS_ID           = (1 << 21),    /* Priv */
+   
+   EFLAGS_IOPL         = 3 << EFLAGS_IOPL_SHIFT,
+   EFLAGS_ARITH        = (EFLAGS_CF | EFLAGS_PF | EFLAGS_AF | EFLAGS_ZF |
+                          EFLAGS_SF | EFLAGS_OF),
+   EFLAGS_USER         = (EFLAGS_CF | EFLAGS_PF | EFLAGS_AF | EFLAGS_ZF |
+                          EFLAGS_SF | EFLAGS_DF | EFLAGS_OF),
+   EFLAGS_PRIV         = (EFLAGS_TF  | EFLAGS_IF  | EFLAGS_IOPL | EFLAGS_NT  |
+                          EFLAGS_RF  | EFLAGS_VM  | EFLAGS_AC   | EFLAGS_VIF |
+                          EFLAGS_VIP | EFLAGS_ID),
+   EFLAGS_ALL          = (EFLAGS_CF | EFLAGS_PF | EFLAGS_AF | EFLAGS_ZF |
+                          EFLAGS_SF | EFLAGS_DF | EFLAGS_OF | EFLAGS_TF |
+                          EFLAGS_IF | EFLAGS_IOPL | EFLAGS_NT | EFLAGS_RF |
+                          EFLAGS_VM | EFLAGS_AC | EFLAGS_VIF | EFLAGS_VIP | 
+                          EFLAGS_ID),
+   EFLAGS_ALL_16       = EFLAGS_ALL & 0xffff,
+   EFLAGS_REAL_32      = (EFLAGS_ALL & ~(EFLAGS_VIP | EFLAGS_VIF | EFLAGS_VM)),
+   EFLAGS_V8086_32     = (EFLAGS_ALL & ~(EFLAGS_VIP | EFLAGS_VIF |
+                                         EFLAGS_VM  | EFLAGS_IOPL)),
+   EFLAGS_REAL_16      = EFLAGS_REAL_32 & 0xffff,
+   EFLAGS_V8086_16     = EFLAGS_V8086_32 & 0xffff,
+   EFLAGS_CLEAR_ON_EXC = (EFLAGS_TF | EFLAGS_VM | EFLAGS_RF | EFLAGS_NT),
+   EFLAGS__4           = 0x7fffffff    /* ensure 4 byte encoding */
+} x86_FLAGS;
+
+/*
+ *   MPX bound configuration registers
+ */
+#define BNDCFG_EN        0x00000001
+#define BNDCFG_BNDPRSV   0x00000002
+#define BNDCFG_RSVD      0x00000ffc
+#define BNDCFG_BDBASE    CONST64U(0xfffffffffffff000)
+
+#endif // ifndef _VM_BASIC_DEFS_H_

vmmon-only/include/x86apic.h

@@ -0,0 +1,354 @@
+/*********************************************************
+ * Copyright (C) 1998,2015-2017 VMware, Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation version 2 and no later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ *********************************************************/
+
+#ifndef _X86APIC_H_
+#define _X86APIC_H_
+
+#define INCLUDE_ALLOW_USERLEVEL
+
+#define INCLUDE_ALLOW_VMMON
+#define INCLUDE_ALLOW_VMK_MODULE
+#define INCLUDE_ALLOW_VMKERNEL
+#define INCLUDE_ALLOW_DISTRIBUTE
+#define INCLUDE_ALLOW_VMCORE
+#include "includeCheck.h"
+
+/*
+ * This file describes all the APIC/IOAPIC register locations and formats
+ * that are defined by the x86 architecture.
+ */
+
+
+/*
+ * APIC registers
+ */
+#define APIC_DEFAULT_ADDRESS      0xfee00000
+#define APIC_ADDRESS_ALIGNMENT    4096
+#define APIC_ADDRESS_MASK         (APIC_ADDRESS_ALIGNMENT-1)
+#define APIC_VERSION_0x11         0x11
+#define XAPIC_VERSION_0x14        0x14
+#define X2APIC_VERSION_0x15       0x15
+#define APIC_MAXLVT_0x4           0x4
+#define XAPIC_MAXLVT_0x5          0x5
+#define XAPIC_MAXLVT_0x6          0x6 // Intel only: Nehalem (1A) and onward
+#define APIC_VERSION_MASK         0xff
+#define APIC_MAX_LVT_MASK         0xff
+#define APIC_MAX_LVT_SHIFT        16
+#define APIC_EXTAPICSPACE_MASK    0x80000000
+#define APIC_INVALID_ID           0xff
+#define X2APIC_INVALID_ID         0xffffffff
+
+#define APICR_ID          0x02
+#define APICR_VERSION     0x03
+#define APICR_TPR         0x08
+#define APICR_APR         0x09
+#define APICR_PPR         0x0a
+#define APICR_EOI         0x0b
+#define APICR_RMTREAD     0x0c
+#define APICR_LDR         0x0d
+#define APICR_DFR         0x0e
+#define APICR_SVR         0x0f
+#define APICR_ISR         0x10
+#define APICR_TMR         0x18
+#define APICR_IRR         0x20
+#define APICR_ESR         0x28
+#define APICR_CMCILVT     0x2f
+#define APICR_ICRLO       0x30
+#define APICR_ICRHI       0x31
+#define APICR_TIMERLVT    0x32
+#define APICR_THERMLVT    0x33
+#define APICR_PCLVT       0x34
+#define APICR_LVT0        0x35
+#define APICR_LVT1        0x36
+#define APICR_ERRLVT      0x37
+#define APICR_INITCNT     0x38
+#define APICR_CURCNT      0x39
+#define APICR_DIVIDER     0x3e
+#define APICR_SELFIPI     0x3f // X2 APIC only
+#define APICR_EXTFEATURE  0x40
+#define APICR_EXTLVT      0x50
+
+#define X2APIC_READONLY_BITMASK \
+   ((CONST64U(1) << APICR_ID)     | (CONST64U(1) << APICR_VERSION) | \
+    (CONST64U(1) << APICR_APR)    | (CONST64U(1) << APICR_PPR)     | \
+    (CONST64U(1) << APICR_CURCNT) | (CONST64U(1) << APICR_ICRHI)   | \
+    (CONST64U(1) << APICR_LDR)    | (CONST64U(1) << APICR_DFR)     | \
+    (MASK64(8) << APICR_ISR)      | (MASK64(8) << APICR_IRR)       | \
+    (MASK64(8) << APICR_TMR))
+
+#define APICR_SIZE             0x540
+
+#define APIC_TPR_RESERVED     0xffffff00
+#define APIC_PR_MASK          0x000000ff
+#define APIC_PR_XMASK         0x000000f0
+#define APIC_PR_YMASK         0x0000000f
+#define APIC_PR_X(_r)         (((_r) & APIC_PR_XMASK) >> 4)
+#define APIC_PR_Y(_r)         (((_r) & APIC_PR_YMASK) >> 0)
+
+#define APIC_SVR_ONES         0x0000000f
+#define APIC_SVR_VECTOR       0x000000ff
+#define APIC_SVR_APICENABLE   0x00000100
+#define APIC_SVR_FOCUSCHECK   0x00000200
+#define APIC_SVR_X2_RESERVED  0xffffee00
+
+#define APIC_LVT_MASK         0x10000
+#define APIC_LVT_DELVMODE_NMI 0x400
+#define APIC_LVT_DELVMODE(_lvt) (_lvt & 0x700)
+#define APIC_LVT_RESET_VALUE  0x00010000
+
+#define APIC_LVT_TIMER_RESERVED 0xfff8ef00
+#define APIC_LVT_ERROR_RESERVED 0xfffeef00 
+#define APIC_LVT_LINT_RESERVED  0xfffe0800
+#define APIC_LVT_OTHER_RESERVED 0xfffee800 // CMCI, PC, THERM LVTs
+
+#define APIC_EXTFEATURE_DEFAULT  0x00040004
+#define APIC_EXTLVT_DEFAULT      0x00010000
+
+/*
+ * Use APIC_MSR_BASEMASK as mask only for reading the MSR_APIC_BASE.
+ * Use guestReservedAPIC as mask when writing MSR_APIC_BASE.
+ */
+#define APIC_MSR_BASEMASK     CONST64U(0x0000fffffffff000)
+#define APIC_MSR_X2APIC_ENABLED 0x00000400
+#define APIC_MSR_ENABLED      0x00000800
+#define APIC_MSR_BSP          0x00000100
+
+#define APIC_VTE_VECTOR_MASK      0x000000ff
+#define APIC_VTE_MODE_FIXED       0x00000000
+#define APIC_VTE_MODE_SMI         0x00000200
+#define APIC_VTE_MODE_NMI         0x00000400
+#define APIC_VTE_MODE_EXTINT      0x00000700
+#define APIC_VTE_MODE_MASK        0x00000700
+#define APIC_VTE_DELVSTATUS       0x00001000
+#define APIC_VTE_PINPOL           0x00002000
+#define APIC_VTE_REMIRR           0x00004000
+#define APIC_VTE_TRIGMODE         0x00008000
+#define APIC_VTE_MASK             0x00010000
+#define APIC_VTE_TMR_ONESHOT      0x00000000
+#define APIC_VTE_TMR_PERIODIC     0x00020000
+#define APIC_VTE_TMR_TSC_DEADLINE 0x00040000
+#define APIC_VTE_TMR_MODE_MASK    0x00070000
+
+/* 
+ * Kernels 2.4.0-test8 and higher have the definition 
+ * APIC_ID_MASK, so we use APIC_ID_BITS to avoid naming conflicts.
+ */
+#define APIC_ID_BITS          0x0f000000
+#define XAPIC_ID_BITS         0xff000000
+#define XAPIC_ID_MASK         0xff000000
+#define X2APIC_ID_BITS        0xffffffff
+#define APIC_ID_SHIFT         24
+#define APIC_LDR_BITS         0xff000000
+#define APIC_LDR_SHIFT        24
+
+#define APIC_DIVIDER_BY_1     0x0000000b
+#define APIC_DIVIDER_RESERVED 0xfffffff4
+
+/* APIC illegal vectors */
+#define APIC_MIN_LEGAL_VECTOR 16
+
+/* APIC delivery modes */
+#define APIC_DELMODE_FIXED    0
+#define APIC_DELMODE_LOWEST   1
+#define APIC_DELMODE_SMI      2
+#define APIC_DELMODE_RSVD     3
+#define APIC_DELMODE_NMI      4
+#define APIC_DELMODE_INIT     5
+#define APIC_DELMODE_STARTUP  6
+#define APIC_DELMODE_EXTINT   7
+
+/* APIC destination modes */
+#define APIC_DESTMODE_PHYS    0
+#define APIC_DESTMODE_LOGICAL 1
+
+/* APIC polarities (active high/low) */
+#define APIC_POLARITY_HIGH    0
+#define APIC_POLARITY_LOW     1
+
+/* APIC trigger types (edge/level) */
+#define APIC_TRIGGER_EDGE     0
+#define APIC_TRIGGER_LEVEL    1
+
+/* APIC destination shorthands */
+#define APIC_DEST_DEST        0
+#define APIC_DEST_LOCAL       1
+#define APIC_DEST_ALL_INC     2
+#define APIC_DEST_ALL_EXC     3
+
+/* APIC physical mode broadcasts */
+#define APIC_DEST_BROADCAST   0x0f
+#define XAPIC_DEST_BROADCAST  0xff
+#define X2APIC_DEST_BROADCAST 0xffffffff
+
+/* APIC IPI Command Register format */
+#define APIC_ICRHI_RESERVED         0x00ffffff
+#define APIC_ICRHI_DEST_MASK        0xff000000
+#define APIC_ICRHI_DEST_OFFSET      24
+#define X2APIC_ICR_DEST_OFFSET      32
+
+#define APIC_ICRLO_RESERVED         0xfff32000
+#define APIC_ICRLO_DEST_MASK        0x000c0000
+#define APIC_ICRLO_DEST_OFFSET      18
+#define APIC_ICRLO_TRIGGER_MASK     0x00008000
+#define APIC_ICRLO_TRIGGER_OFFSET   15
+#define APIC_ICRLO_LEVEL_MASK       0x00004000
+#define APIC_ICRLO_LEVEL_OFFSET     14
+#define APIC_ICRLO_STATUS_MASK      0x00001000
+#define APIC_ICRLO_STATUS_OFFSET    12
+#define APIC_ICRLO_DESTMODE_MASK    0x00000800
+#define APIC_ICRLO_DESTMODE_OFFSET  11
+#define APIC_ICRLO_DELMODE_MASK     0x00000700
+#define APIC_ICRLO_DELMODE_OFFSET   8
+#define APIC_ICRLO_VECTOR_MASK      0x000000ff
+#define APIC_ICRLO_VECTOR_OFFSET    0
+
+/* x2APIC Logical ID fields */ 
+#define X2APIC_LDR_BITVEC_MASK      0x0000ffff
+#define X2APIC_LDR_CLUSTER_MASK     0xffff0000
+#define X2APIC_LDR_CLUSTER_SHIFT    16
+
+/* APIC error register bits */ 
+#define APIC_ERR_ILL_REG         (1<<7)
+#define APIC_REC_ILL_VEC         (1<<6)
+#define APIC_SENT_ILL_VEC        (1<<5)
+#define APIC_ERR_REDIR_IPI       (1<<4) // X2APIC
+
+#define X2APIC_SELFIPI_RESERVED     0xffffff00
+
+/*
+ * APIC register accessors
+ */
+#define APIC_VERSION_REG(_apic)  (_apic[APICR_VERSION][0])
+#define APIC_MAX_LVT(_apic)      ((APIC_VERSION_REG(_apic) >> \
+                                   APIC_MAX_LVT_SHIFT) & APIC_MAX_LVT_MASK)
+
+#define APIC_LVT_ISMASKED(_lvt)  (_lvt & APIC_LVT_MASK)
+#define APIC_LVT_VECTOR(_lvt)    (_lvt & 0xff)
+
+#define APIC_SPINT_REG(_apic)    (_apic[APICR_SVR][0])
+#define APIC_TIMER_REG(_apic)    (_apic[APICR_TIMERLVT][0])
+#define APIC_THERM_REG(_apic)    (_apic[APICR_THERMLVT][0])
+#define APIC_PC_REG(_apic)       (_apic[APICR_PCLVT][0])
+#define APIC_LINT0_REG(_apic)    (_apic[APICR_LVT0][0])
+#define APIC_LINT1_REG(_apic)    (_apic[APICR_LVT1][0])
+#define APIC_ERR_REG(_apic)      (_apic[APICR_ERRLVT][0])
+#define APIC_INITCNT_REG(_apic)  (_apic[APICR_INITCNT][0])
+#define APIC_CURCNT_REG(_apic)   (_apic[APICR_CURCNT][0])
+
+#define APIC_SPINT_VECTOR(_apic) (APIC_SPINT_REG(_apic) & 0xff)
+#define APIC_TIMER_VECTOR(_apic) (APIC_TIMER_REG(_apic) & 0xff)
+#define APIC_PC_VECTOR(_apic)    (APIC_PC_REG(_apic) & 0xff)
+#define APIC_LINT0_VECTOR(_apic) (APIC_LINT0_REG(_apic) & 0xff)
+#define APIC_LINT1_VECTOR(_apic) (APIC_LINT1_REG(_apic) & 0xff)
+#define APIC_ERR_VECTOR(_apic)   (APIC_ERR_REG(_apic) & 0xff)
+
+
+/*
+ * I/O APIC registers
+ */
+#define IOAPIC_DEFAULT_ADDRESS    0xfec00000
+#define IOAPIC_ADDRESS_ALIGNMENT  1024
+#define IOAPIC_ADDRESS_MASK       (IOAPIC_ADDRESS_ALIGNMENT-1)
+#define IOAPIC_UVERSIONMASK       0xff
+
+#define IOAPICID              0x00
+#define IOAPICVER             0x01
+#define IOAPICARB             0x02
+#define IOREDTBL_FIRST        0x10
+#define IOREDTBL_LAST         0x3f
+
+#define IO_APIC_TIMER_PIN               2
+#define IO_APIC_RTC_PIN                 8
+
+#define IO_APIC_REG0_RES2_MASK          0x00FFFFFF
+#define IO_APIC_REG0_RES2_OFFSET        0
+#define IO_APIC_REG0_RES1_MASK          0xF0000000
+#define IO_APIC_REG0_RES1_OFFSET        28
+
+#define IO_APIC_REG1_VERSION_MASK       0x000000FF
+#define IO_APIC_REG1_VERSION_OFFSET     0
+#define IO_APIC_REG1_RES2_MASK          0x0000FF00
+#define IO_APIC_REG1_RES2_OFFSET        8
+#define IO_APIC_REG1_ENTRIES_MASK       0x00FF0000
+#define IO_APIC_REG1_ENTRIES_OFFSET     16
+#define IO_APIC_REG1_RES1_MASK          0xFF000000
+#define IO_APIC_REG1_RES1_OFFSET        24
+
+#define IO_APIC_REG2_RES1_MASK          0x00FFFFFF
+#define IO_APIC_REG2_RES1_OFFSET        0
+#define IO_APIC_REG2_RES2_MASK          0xF0000000
+#define IO_APIC_REG2_RES2_OFFSET        28
+
+#define IO_APIC_ROUTE_VECTOR_MASK       0x000000FF
+#define IO_APIC_ROUTE_VECTOR_OFFSET     0
+
+#define IO_APIC_INTMASK_MASK            0x00010000
+#define IO_APIC_INTMASK_OFFSET          16
+
+#define IO_APIC_DELMODE_MASK            0x00000700
+#define IO_APIC_DELMODE_OFFSET          8
+
+#define IO_APIC_DESTMODE_MASK           0x00000800
+#define IO_APIC_DESTMODE_OFFSET         11
+
+#define IO_APIC_POLARITY_MASK           0x00002000
+#define IO_APIC_POLARITY_OFFSET         13
+
+#define IO_APIC_TRIGGER_MASK            0x00008000
+#define IO_APIC_TRIGGER_OFFSET          15
+
+#define IO_APIC_DEST_MASK               0xff000000
+#define IO_APIC_DEST_OFFSET             24
+
+#define IO_APIC_REG0_RES1(_reg) \
+   ((_reg & IO_APIC_REG0_RES1_MASK) >> IO_APIC_REG0_RES1_OFFSET)
+#define IO_APIC_REG0_RES2(_reg) \
+   ((_reg & IO_APIC_REG0_RES2_MASK) >> IO_APIC_REG0_RES2_OFFSET)
+#define IO_APIC_REG1_RES1(_reg) \
+   ((_reg & IO_APIC_REG1_RES1_MASK) >> IO_APIC_REG1_RES1_OFFSET)
+#define IO_APIC_REG1_RES2(_reg) \
+   ((_reg & IO_APIC_REG1_RES2_MASK) >> IO_APIC_REG1_RES2_OFFSET)
+#define IO_APIC_REG2_RES1(_reg) \
+   ((_reg & IO_APIC_REG2_RES1_MASK) >> IO_APIC_REG2_RES1_OFFSET)
+#define IO_APIC_REG2_RES2(_reg) \
+   ((_reg & IO_APIC_REG2_RES2_MASK) >> IO_APIC_REG2_RES2_OFFSET)
+#define IO_APIC_VERSION(_reg) \
+   ((_reg & IO_APIC_REG1_VERSION_MASK) >> IO_APIC_REG1_VERSION_OFFSET)
+#define IO_APIC_ENTRIES(_reg) \
+   ((_reg & IO_APIC_REG1_ENTRIES_MASK) >> IO_APIC_REG1_ENTRIES_OFFSET)
+#define IO_APIC_ENTRY_VECTOR(_entry) \
+   ((_entry & IO_APIC_ROUTE_VECTOR_MASK) >> IO_APIC_ROUTE_VECTOR_OFFSET)
+#define IO_APIC_ISMASKED(_entry) \
+   ((_entry & IO_APIC_INTMASK_MASK) >> IO_APIC_INTMASK_OFFSET)
+#define IO_APIC_ENTRY_DELMODE(_entry) \
+   ((_entry & IO_APIC_DELMODE_MASK) >> IO_APIC_DELMODE_OFFSET)
+#define IO_APIC_ENTRY_DESTMODE(_entry) \
+   ((_entry & IO_APIC_DESTMODE_MASK) >> IO_APIC_DESTMODE_OFFSET)
+#define IO_APIC_ENTRY_POLARITY(_entry) \
+   ((_entry & IO_APIC_POLARITY_MASK) >> IO_APIC_POLARITY_OFFSET)
+#define IO_APIC_ENTRY_TRIGGER(_entry) \
+   ((_entry & IO_APIC_TRIGGER_MASK) >> IO_APIC_TRIGGER_OFFSET)
+#define IO_APIC_ENTRY_DEST(_entry) \
+   ((_entry & IO_APIC_DEST_MASK) >> IO_APIC_DEST_OFFSET)
+
+//
+// We emulate an IOAPIC with 24 redirection registers
+//
+#define IOAPIC_NUM_REDIR_REGS 24
+
+#endif // _X86APIC_H_